measurement to the server. On systems where the Gauger client is not installed, the call fails silently so as not to disrupt normal operations in any way. The syntax of the Gauger client command-line tool is as follows:
$ gauger [-i ID] -c CATEGORY -n NAME \ -u UNIT -d DATA
Here, NAME is the name of the met-ric, and DATA is any floating-point value that represents the performance measurement. UNIT is a string that describes the unit of the value, for example, ms/request. CATEGORY is a string used to group multiple perfor-mance metrics by subsystem. We
recommend using the name of the subsystem or module here, especially for larger projects.
Revision Numbers
Gauger can autodetect the current revision of the project if the bench-mark is executed in a directory that was checked out from a supported Version Control System (VCS). The supported VCSes are Subversion, Git, hg, bazaar, monotone and GNU arch.
For distributed VCSes that do not pro-vide an ascending revision numbering system (like Git), Gauger uses the number of commits to the repository.
In this case, all execution hosts used for benchmarking should use the same origin to keep the data consistent. If the project uses an unsupported VCS or if the benchmark is executed out-side the tree, Gauger needs to know which (revision) number to use for the FEATURE Performance Regression Monitoring with Gauger
Listing 2. The GAUGER macro makes it easy to integrate Gauger with C code.
Note that the code does not need to be linked against any additional libraries (other than libc).
#include <gauger.h>
#include <time.h>
int main() {
time_t start = time (NULL);
do_test ();
time_t delay = time (NULL) - start;
GAUGER ("subsystem", "execution time for f", delay, "s");
return 0;
}
Listing 3. A simple static method call, leading to a single line of Java code, can be used to invoke Gauger from Java.
import static org.gnunet.gauger.Gauger.gauger;
class HelloGauger {
public static void main(String[] args) { gauger ("subsystem", "Speed",
42 /* value */, "kb/s");
} }
x-axis. The --id ID option is used to supply the revision number in this case.
Some projects may want to use an internal version number or a timestamp instead of a revision number generated by their VCS. The only restriction imposed on the numbers used is that Gauger’s regression monitoring
assumes a linear progression of devel-opment. For projects with multiple branches under active development, different Gauger servers should be set up for each branch.
Language Bindings
Gauger ships with bindings for various languages (see, for example, Listings 2, 3, 4 and 5) to make integration easy. The resulting binaries do not depend on a Gauger client installation on the target system. The bindings should be integrated with the main project and, as mentioned before, simply do nothing when invoked if the Gauger client is not installed or not configured properly.
The JavaScript binding is unusual.
Because JavaScript cannot access the local filesystem to read the
configura-tion file, the login data must be stored in a cookie in advance. The login page on the Gauger Web site, which usually is used to set up new accounts for execution hosts, can be used to set the respective login cookie in the browser. Access to the source code’s repository also is not possible from JavaScript, so the revision number must be supplied explicitly to the GAUGER call. Typically, the current revision number is obtained on the server side. For example, PHP code can be used to obtain the number from the VCS, or on-commit trigger
WWW.LINUXJOURNAL.COM SEPTEMBER 2011 | 73
Listing 4. Example Code for Invoking Gauger from Python
from gauger import GAUGER
GAUGER("CAT", "NAME", 42, "kb/s")
Listing 5. The browser must be regis-tered with the Gauger Web site before Gauger can be invoked from JavaScript.
Once the login cookie is set, the main difference from other languages is that the JavaScript code must supply its revision number explicitly.
[...]
<script type="text/javascript"
src="http://www.example.com/gauger.js">
</script>
[...]
var rev = $Revision$;
/* or */
var rev = <?php echo get_revision() ?>;
var performance = do_test();
GAUGER ('CAT', 'NAME',
performance, 'kb/s', rev)" />
[...]
functions provided by the VCS could be used to insert the number into the source code.
Selecting Proper Units
Gauger provides developers complete freedom with respect to the names, values and units of the performance metrics to be monitored. So, how do you choose a good unit? Naturally, part of the unit always is dictated by the kind of performance characteristic you are interested in—for example, execution time (seconds) or space consumption (bytes). However, generally it’s a good idea always to relate this basic unit to the amount of work performed as part of the unit given to Gauger.
For example, suppose a benchmark measures the execution time for 100 GET requests. In this case, it is better to choose a name “GET request perfor-mance” with unit “requests/second”
(and log the execution time divided by 100) instead of the name “Time for 100 GET requests” with unit “seconds”.
The reason for this is it’s quite possible the benchmark will be adjusted in the future—for example, to run 1,000 GET requests. If performance is logged as
“requests/second”, such a change would then not require any changes to
the name of the tracked metric. As a result, the performance regression analysis can continue to track the metric in the same curve.
Additionally, Gauger allows different results to be compared by adding new metrics to existing plots. If the new metric uses the same unit as the old one, they will use the same y-axis;
otherwise, the new one will be plotted against a second y-axis on the right side of the plot. This limits the number of units per plot to two. We recom-mend using the same units where applicable (for example, no mixing of
“seconds” and “milliseconds”) to make comparative analysis easier.
Related Projects
Gauger does not include for support for actually automatically running the benchmarks on various systems.
However, this is not a drawback, because an excellent tool for that purpose exists in the form of Buildbot.
Buildbot typically is used for regression testing and, hence, is by itself not suitable for identifying performance regressions. Nevertheless, Buildbot requires a similar network setup—that is, clients that run the tests connect to a public server. This makes it trivial to FEATURE Performance Regression Monitoring with Gauger