1README for libm-test math test suite 2==================================== 3 4The libm-test math test suite tests a number of function points of 5math functions in the GNU C library. The following sections contain a 6brief overview. Please note that the test drivers and the Python 7script "gen-libm-test.py" have some options. A full list of options 8is available with --help (for the test drivers) and -h for 9"gen-libm-test.py". 10 11 12What is tested? 13=============== 14The tests just evaluate the functions at specified points and compare 15the results with precomputed values and the requirements of the ISO 16C99 standard. 17 18Besides testing the special values mandated by IEEE 754 (infinities, 19NaNs and minus zero), some more or less random values are tested. 20 21Files that are part of libm-test 22================================ 23 24The main files are "libm-test-<func>.inc". They are independent of 25the target platform and the specific real floating type and format and 26contain placeholder test "templates" for math functions defined in 27libm. These files, along with generated files named 28"auto-libm-test-out-<func>", are preprocessed by the Python script 29"gen-libm-test.py" to expand the templates and produce a set of test 30cases for each math function that are specific to the target platform 31but still independent of the real floating type. The results of the 32processing are "libm-test-<func>.c" and a file "libm-test-ulps.h" with 33platform specific deltas by which the actual math function results may 34deviate from the expected results and still be considered correct. 35 36The test drivers "test-double-<func>.c", "test-float-<func>.c", and 37"test-ldouble-<func>.c", generated by the Makefile, test the normal 38double, float and long double implementation of libm. Each driver 39selects the desired real floating type to exercise the math functions 40to test with (float, double, or long double) by defining a small set 41of macros just before including the generic "libm-test.c" file. Each 42driver is compiled into a single executable test program with the 43corresponding name. 44 45As mentioned above, the "gen-libm-test.py" script looks for a file 46named "libm-test-ulps" in the platform specific sysdep directory (or 47its fpu or nofpu subdirectory) and for each variant (real floating 48type and rounding mode) of every tested function reads from it the 49maximum difference expressed as Units of Least Precision (ULP) the 50actual result of the function may deviate from the expected result 51before it's considered incorrect. 52 53The "auto-libm-test-out-<func>" files contain sets of test cases to 54exercise, the conditions under which to exercise each, and the 55expected results. The files are generated by the 56"gen-auto-libm-tests" program from the "auto-libm-test-in" file. See 57the comments in gen-auto-libm-tests.c for details about the content 58and format of the -in and -out files. 59 60How can I generate "libm-test-ulps"? 61==================================== 62 63To automatically generate a new "libm-test-ulps" run "make regen-ulps". 64This generates the file "math/NewUlps" in the build directory. The file 65contains the sorted results of all the tests. You can use the "NewUlps" 66file as the machine's updated "libm-test-ulps" file. Copy "NewUlps" to 67"libm-test-ulps" in the appropriate machine sysdep directory. Verify 68the changes, post your patch, and check it in after review. 69 70To manually generate a new "libm-test-ulps" file, first remove "ULPs" 71file in the current directory, then you can execute for example: 72 ./testrun.sh math/test-double -u --ignore-max-ulp=yes 73This generates a file "ULPs" with all double ULPs in it, ignoring any 74previously calculated ULPs, and running with the newly built dynamic 75loader and math library (assumes you didn't install your build). Now 76generate the ULPs for all other formats, the tests will be appending the 77data to the "ULPs" file. As final step run "gen-libm-test.py" with the 78file as input and ask to generate a pretty printed output in the file 79"NewUlps": 80 gen-libm-test.py -u ULPs -n NewUlps 81Copy "NewUlps" to "libm-test-ulps" in the appropriate machine sysdep 82directory. 83 84Note that the test drivers have an option "-u" to output an unsorted 85list of all epsilons that the functions have. The output can be read 86in directly but it's better to pretty print it first. 87"gen-libm-test.py" has an option to generate a pretty-printed and 88sorted new ULPs file from the output of the test drivers. 89 90Contents of libm-test-ulps 91========================== 92 93Since libm-test-ulps can be generated automatically, just a few notes. 94The file contains lines for maximal errors of single functions, like: 95 96Function "yn": 97double: 6 98 99The keywords are float, double, and ldouble. 100 101Adding tests to libm-test-<func>.inc 102==================================== 103 104The tests are evaluated by a set of special test macros. The macros 105start with "TEST_" followed by a specification the input values, an 106underscore and a specification of the output values. As an example, 107the test macro for a function with input of type FLOAT (FLOAT is 108either float, double, long double) and output of type FLOAT is 109"TEST_f_f". The macro's parameter are the name of the function, the 110input parameter, output parameter and optionally one exception 111parameter. 112 113The accepted parameter types are: 114- "f" for FLOAT 115- "j" for long double. 116- "a" for ARG_FLOAT, the argument type for narrowing functions. 117- "b" for boolean - just tests if the output parameter evaluates to 0 118 or 1 (only for output). 119- "c" for complex. This parameter needs two values, first the real, 120 then the imaginary part. 121- "i" for int. 122- "l" for long int. 123- "L" for long long int. 124- "u" for unsigned int. 125- "M" for intmax_t. 126- "U" for uintmax_t. 127- "p" for an argument (described in the previous character) passed 128 through a pointer rather than directly. 129- "F" for the address of a FLOAT (only as input parameter) 130- "I" for the address of an int (only as input parameter) 131- "1" for an additional output (either output through a pointer passed 132 as an argument, or to a global variable such as signgam). 133 134How to read the test output 135=========================== 136 137Running each test on its own at the default level of verbosity will 138print on stdout a line describing the implementation of math functions 139exercised by the test (float, double, or long double). This is then 140followed by the details of test failures (if any). The output concludes 141by a summary listing the number of test cases exercised and the number 142of test failures uncovered. 143 144For each test failure (and for each test case at higher levels of 145verbosity), the output contains the name of the function under test 146and its arguments or conditions that triggered the failure. Note 147that the name of the function in the output need not correspond 148exactly to the name of the math function actually invoked. For example, 149the output will refer to the "acos" function even if the actual function 150under test is acosf (for the float version) or acosl (for the long 151double version). Also note that the function arguments may be shown 152in either the decimal or the hexadecimal floating point format which 153may or may not correspond to the format used in the auto-libm-test-in 154file. Besides the name of the function, for each test failure the 155output contains the actual and expected results and the difference 156between the two, printed in both the decimal and hexadecimal 157floating point format, and the ULP and maximum ULP for the test 158case. 159