1Using the glibc microbenchmark suite
2====================================
3
4The glibc microbenchmark suite automatically generates code for specified
5functions, builds and calls them repeatedly for given inputs to give some
6basic performance properties of the function.
7
8Running the benchmark:
9=====================
10
11The benchmark needs python 2.7 or later in addition to the
12dependencies required to build the GNU C Library.  One may run the
13benchmark by invoking make as follows:
14
15  $ make bench
16
17This runs each function for 10 seconds and appends its output to
18benchtests/bench.out.  To ensure that the tests are rebuilt, one could run:
19
20  $ make bench-clean
21
22The duration of each test can be configured setting the BENCH_DURATION variable
23in the call to make.  One should run `make bench-clean' before changing
24BENCH_DURATION.
25
26  $ make BENCH_DURATION=1 bench
27
28The benchmark suite does function call measurements using architecture-specific
29high precision timing instructions whenever available.  When such support is
30not available, it uses clock_gettime (CLOCK_MONOTONIC).  One can force the
31benchmark to use clock_gettime by invoking make as follows:
32
33  $ make USE_CLOCK_GETTIME=1 bench
34
35Again, one must run `make bench-clean' before changing the measurement method.
36
37On x86 processors, RDTSCP instruction provides more precise timing data
38than RDTSC instruction.  All x86 processors since 2010 support RDTSCP
39instruction.  One can force the benchmark to use RDTSCP by invoking make
40as follows:
41
42  $ make USE_RDTSCP=1 bench
43
44One must run `make bench-clean' before changing the measurement method.
45
46Running benchmarks on another target:
47====================================
48
49If the target where you want to run benchmarks is not capable of building the
50code or you're cross-building, you could build and execute the benchmark in
51separate steps.  On the build system run:
52
53  $ make bench-build
54
55and then copy the source and build directories to the target and run the
56benchmarks from the build directory as usual:
57
58  $ make bench
59
60make sure the copy preserves timestamps by using either rsync or scp -p
61otherwise the above command may try to build the benchmark again.  Benchmarks
62that require generated code to be executed during the build are skipped when
63cross-building.
64
65Building benchmarks as static executables:
66=========================================
67
68To build benchmarks as static executables, on the build system, run:
69
70  $ make STATIC-BENCHTESTS=yes bench-build
71
72You can copy benchmark executables to another machine and run them
73without copying the source nor build directories.
74
75Running subsets of benchmarks:
76==============================
77
78To run only a subset of benchmarks, one may invoke make as follows
79
80  $ make bench BENCHSET="bench-pthread bench-math malloc-thread"
81
82where BENCHSET may be a space-separated list of the following values:
83
84    bench-math
85    bench-pthread
86    bench-string
87    hash-benchset
88    malloc-thread
89    math-benchset
90    stdio-common-benchset
91    stdlib-benchset
92    string-benchset
93    wcsmbs-benchset
94
95Adding a function to benchtests:
96===============================
97
98If the name of the function is `foo', then the following procedure should allow
99one to add `foo' to the bench tests:
100
101- Append the function name to the bench variable in the Makefile.
102
103- Make a file called `foo-inputs` to provide the definition and input for the
104  function.  The file should have some directives telling the parser script
105  about the function and then one input per line.  Directives are lines that
106  have a special meaning for the parser and they begin with two hashes '##'.
107  The following directives are recognized:
108
109  - args: This should be assigned a colon separated list of types of the input
110    arguments.  This directive may be skipped if the function does not take any
111    inputs.  One may identify output arguments by nesting them in <>.  The
112    generator will create variables to get outputs from the calling function.
113  - ret: This should be assigned the type that the function returns.  This
114    directive may be skipped if the function does not return a value.
115  - includes: This should be assigned a comma-separated list of headers that
116    need to be included to provide declarations for the function and types it
117    may need (specifically, this includes using "#include <header>").
118  - include-sources: This should be assigned a comma-separated list of source
119    files that need to be included to provide definitions of global variables
120    and functions (specifically, this includes using "#include "source").
121    See pthread_once-inputs and pthreads_once-source.c for an example of how
122    to use this to benchmark a function that needs state across several calls.
123  - init: Name of an initializer function to call to initialize the benchtest.
124  - name: See following section for instructions on how to use this directive.
125
126  Lines beginning with a single hash '#' are treated as comments.  See
127  pow-inputs for an example of an input file.
128
129Multiple execution units per function:
130=====================================
131
132Some functions have distinct performance characteristics for different input
133domains and it may be necessary to measure those separately.  For example, some
134math functions perform computations at different levels of precision (64-bit vs
135240-bit vs 768-bit) and mixing them does not give a very useful picture of the
136performance of these functions.  One could separate inputs for these domains in
137the same file by using the `name' directive that looks something like this:
138
139  ##name: 240bits
140
141All inputs after the ##name: 240bits directive and until the next `name'
142directive (or the end of file) are part of the "240bits" benchmark and
143will be output separately in benchtests/bench.out.  See the pow-inputs file
144for an example of what such a partitioned input file would look like.
145
146It is also possible to measure latency and reciprocal throughput of a
147(partial) trace extracted from a real workload.  In this case the whole trace
148is iterated over multiple times rather than repeating every input multiple
149times.  This can be done via:
150
151  ##name: workload-<name>
152
153where <name> is simply used to distinguish between different traces in the
154same file.  To create such a trace, you can simply extract using printf()
155values uses for a specific application, or generate random values in some
156interval.  See the expf-inputs file for an example of this workload mechanism.
157
158Benchmark Sets:
159==============
160
161In addition to standard benchmarking of functions, one may also generate
162custom outputs for a set of functions.  This is currently used by string
163function benchmarks where the aim is to compare performance between
164implementations at various alignments and for various sizes.
165
166To add a benchset for `foo':
167
168- Add `foo' to the benchset variable.
169- Write your bench-foo.c that prints out the measurements to stdout.
170- On execution, a bench-foo.out is created in $(objpfx) with the contents of
171  stdout.
172
173Reading String Benchmark Results:
174================================
175
176Some of the string benchmark results are now in JSON to make it easier to read
177in scripts.  Use the benchtests/compare_strings.py script to show the results
178in a tabular format, generate graphs and more. Run
179
180    benchtests/scripts/compare_strings.py -h
181
182for usage information.
183