1#!/usr/bin/python3 2# Generate testcase files and Makefile fragments for DSO sorting test 3# Copyright (C) 2021-2022 Free Software Foundation, Inc. 4# This file is part of the GNU C Library. 5# 6# The GNU C Library is free software; you can redistribute it and/or 7# modify it under the terms of the GNU Lesser General Public 8# License as published by the Free Software Foundation; either 9# version 2.1 of the License, or (at your option) any later version. 10# 11# The GNU C Library is distributed in the hope that it will be useful, 12# but WITHOUT ANY WARRANTY; without even the implied warranty of 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14# Lesser General Public License for more details. 15# 16# You should have received a copy of the GNU Lesser General Public 17# License along with the GNU C Library; if not, see 18# <http://www.gnu.org/licenses/>. 19 20"""Generate testcase files and Makefile fragments for DSO sorting test 21 22This script takes a small description string language, and generates 23testcases for displaying the ELF dynamic linker's dependency sorting 24behavior, allowing verification. 25 26Testcase descriptions are semicolon-separated description strings, and 27this tool generates a testcase from the description, including main program, 28associated modules, and Makefile fragments for including into elf/Makefile. 29 30This allows automation of what otherwise would be very laborous manual 31construction of complex dependency cases, however it must be noted that this 32is only a tool to speed up testcase construction, and thus the generation 33features are largely mechanical in nature; inconsistencies or errors may occur 34if the input description was itself erroneous or have unforeseen interactions. 35 36The format of the input test description files are: 37 38 # Each test description has a name, lines of description, 39 # and an expected output specification. Comments use '#'. 40 testname1: <test-description-line> 41 output: <expected-output-string> 42 43 # Tests can be marked to be XFAIL by using 'xfail_output' instead 44 testname2: <test-description-line> 45 xfail_output: <expected-output-string> 46 47 # A default set of GLIBC_TUNABLES tunables can be specified, for which 48 # all following tests will run multiple times, once for each of the 49 # GLIBC_TUNABLES=... strings set by the 'tunable_option' command. 50 tunable_option: <glibc-tunable-string1> 51 tunable_option: <glibc-tunable-string2> 52 53 # Test descriptions can use multiple lines, which will all be merged 54 # together, so order is not important. 55 testname3: <test-description-line> 56 <test-description-line> 57 <test-description-line> 58 ... 59 output: <expected-output-string> 60 61 # 'testname3' will be run and compared two times, for both 62 # GLIBC_TUNABLES=<glibc-tunable-string1> and 63 # GLIBC_TUNABLES=<glibc-tunable-string2>. This can be cleared and reset by the 64 # 'clear_tunables' command: 65 clear_tunables 66 67 # Multiple expected outputs can also be specified, with an associated 68 # tunable option in (), which multiple tests will be run with each 69 # GLIBC_TUNABLES=... option tried. 70 testname4: 71 <test-description-line> 72 ... 73 output(<glibc-tunable-string1>): <expected-output-string-1> 74 output(<glibc-tunable-string2>): <expected-output-string-2> 75 # Individual tunable output cases can be XFAILed, though note that 76 # this will have the effect of XFAILing the entire 'testname4' test 77 # in the final top-level tests.sum summary. 78 xfail_output(<glibc-tunable-string3>): <expected-output-string-3> 79 80 # When multiple outputs (with specific tunable strings) are specified, 81 # these take priority over any active 'tunable_option' settings. 82 83 # When a test is meant to be placed under 'xtests' (not run under 84 # "make check", but only when "make xtests" is used), the testcase name can be 85 # declared using 'xtest(<test-name>)': 86 ... 87 xtest(test-too-big1): <test-description> 88 output: <expected-output-string> 89 ... 90 91 # Do note that under current elf/Makefile organization, for such a xtest case, 92 # while the test execution is only run under 'make xtests', the associated 93 # DSOs are always built even under 'make check'. 94 95On the description language used, an example description line string: 96 97 a->b!->[cdef];c=>g=>h;{+c;%c;-c}->a 98 99Each identifier represents a shared object module, currently sequences of 100letters/digits are allowed, case-sensitive. 101 102All such shared objects have a constructor/destructor generated for them 103that emits its name followed by a '>' for constructors, and '<' followed by 104its name for destructors, e.g. if the name is 'obj1', then "obj1>" and "<obj1" 105is printed by its constructor/destructor respectively. 106 107The -> operator specifies a link time dependency, these can be chained for 108convenience (e.g. a->b->c->d). 109 110The => operator creates a call-reference, e.g. for a=>b, an fn_a() function 111is created inside module 'a', which calls fn_b() in module 'b'. 112These module functions emit 'name()' output in nested form, 113e.g. a=>b emits 'a(b())' 114 115For single character object names, square brackets [] in the description 116allows specifying multiple objects; e.g. a->[bcd]->e is equivalent to 117 a->b->e;a->c->e;a->d->e 118 119The () parenthesis construct with space separated names is also allowed for 120specifying objects. For names with integer suffixes a range can also be used, 121e.g. (foo1 bar2-5), specifies DSOs foo1, bar2, bar2, bar3, bar4, bar5. 122 123A {} construct specifies the main test program, and its link dependencies 124are also specified using ->. Inside {}, a few ;-separated constructs are 125allowed: 126 +a Loads module a using dlopen(RTLD_LAZY|RTLD_GLOBAL) 127 ^a Loads module a using dlopen(RTLD_LAZY) 128 %a Use dlsym() to load and call fn_a() 129 @a Calls fn_a() directly. 130 -a Unloads module a using dlclose() 131 132The generated main program outputs '{' '}' with all output from above 133constructs in between. The other output before/after {} are the ordered 134constructor/destructor output. 135 136If no {} construct is present, a default empty main program is linked 137against all objects which have no dependency linked to it. e.g. for 138'[ab]->c;d->e', the default main program is equivalent to '{}->[abd]' 139 140Sometimes for very complex or large testcases, besides specifying a 141few explicit dependencies from main{}, the above default dependency 142behavior is still useful to automatically have, but is turned off 143upon specifying a single explicit {}->dso_name. 144In this case, add {}->* to explicitly add this generation behavior: 145 146 # Main program links to 'foo', and all other objects which have no 147 # dependency linked to it. 148 {}->foo,{}->* 149 150Note that '*' works not only on main{}, but can be used as the 151dependency target of any object. Note that it only works as a target, 152not a dependency source. 153 154The '!' operator after object names turns on permutation of its 155dependencies, e.g. while a->[bcd] only generates one set of objects, 156with 'a.so' built with a link line of "b.so c.so d.so", for a!->[bcd] 157permutations of a's dependencies creates multiple testcases with 158different link line orders: "b.so c.so d.so", "c.so b.so d.so", 159"b.so d.so c.so", etc. Note that for a <test-name> specified on 160the script command-line, multiple <test-name_1>, <test-name_2>, etc. 161tests will be generated (e.g. for a!->[bc]!->[de], eight tests with 162different link orders for a, b, and c will be generated) 163 164It is possible to specify the ELF soname field for an object or the 165main program: 166 # DSO 'a' will be linked with the appropriate -Wl,-soname=x setting 167 a->b->c;soname(a)=x 168 # The the main program can also have a soname specified 169 soname({})=y 170 171This can be used to test how ld.so behaves when objects and/or the 172main program have such a field set. 173 174 175Strings Output by Generated Testcase Programs 176 177The text output produced by a generated testcase consists of three main 178parts: 179 1. The constructors' output 180 2. Output from the main program 181 3. Destructors' output 182 183To see by example, a simple test description "a->b->c" generates a testcase 184that when run, outputs: "c>b>a>{}<a<b<c" 185 186Each generated DSO constructor prints its name followed by a '>' character, 187and the "c>b>a" part above is the full constructor output by all DSOs, the 188order indicating that DSO 'c', which does not depend on any other DSO, has 189its constructor run first, followed by 'b' and then 'a'. 190 191Destructor output for each DSO is a '<' character followed by its name, 192reflecting its reverse nature of constructors. In the above example, the 193destructor output part is "<a<b<c". 194 195The middle "{}" part is the main program. In this simple example, nothing 196was specified for the main program, so by default it is implicitly linked 197to the DSO 'a' (with no other DSOs depending on it) and only prints the 198brackets {} with no actions inside. 199 200To see an example with actions inside the main program, lets see an example 201description: c->g=>h;{+c;%c;-c}->a->h 202 203This produces a testcase, that when executed outputs: 204 h>a>{+c[g>c>];%c();-c[<c<g];}<a<h 205 206The constructor and destructor parts display the a->h dependency as expected. 207Inside the main program, the "+c" action triggers a dlopen() of DSO 'c', 208causing another chain of constructors "g>c>" to be triggered. Here it is 209displayed inside [] brackets for each dlopen call. The same is done for "-c", 210a dlclose() of 'c'. 211 212The "%c" output is due to calling to fn_c() inside DSO 'c', this comprises 213of two parts: the '%' character is printed by the caller, here it is the main 214program. The 'c' character is printed from inside fn_c(). The '%' character 215indicates that this is called by a dlsym() of "fn_c". A '@' character would 216mean a direct call (with a symbol reference). These can all be controlled 217by the main test program constructs documented earlier. 218 219The output strings described here is the exact same form placed in 220test description files' "output: <expected output>" line. 221""" 222 223import sys 224import re 225import os 226import subprocess 227import argparse 228from collections import OrderedDict 229import itertools 230 231# BUILD_GCC is only used under the --build option, 232# which builds the generated testcase, including DSOs using BUILD_GCC. 233# Mainly for testing purposes, especially debugging of this script, 234# and can be changed here to another toolchain path if needed. 235build_gcc = "gcc" 236 237def get_parser(): 238 parser = argparse.ArgumentParser("") 239 parser.add_argument("description", 240 help="Description string of DSO dependency test to be " 241 "generated (see script source for documentation of " 242 "description language), either specified here as " 243 "command line argument, or by input file using " 244 "-f/--description-file option", 245 nargs="?", default="") 246 parser.add_argument("test_name", 247 help="Identifier for testcase being generated", 248 nargs="?", default="") 249 parser.add_argument("--objpfx", 250 help="Path to place generated files, defaults to " 251 "current directory if none specified", 252 nargs="?", default="./") 253 parser.add_argument("-m", "--output-makefile", 254 help="File to write Makefile fragment to, defaults to " 255 "stdout when option not present", 256 nargs="?", default="") 257 parser.add_argument("-f", "--description-file", 258 help="Input file containing testcase descriptions", 259 nargs="?", default="") 260 parser.add_argument("--build", help="After C testcase generated, build it " 261 "using gcc (for manual testing purposes)", 262 action="store_true") 263 parser.add_argument("--debug-output", 264 help="Prints some internal data " 265 "structures; used for debugging of this script", 266 action="store_true") 267 return parser 268 269# Main script starts here. 270cmdlineargs = get_parser().parse_args() 271test_name = cmdlineargs.test_name 272description = cmdlineargs.description 273objpfx = cmdlineargs.objpfx 274description_file = cmdlineargs.description_file 275output_makefile = cmdlineargs.output_makefile 276makefile = "" 277default_tunable_options = [] 278 279current_input_lineno = 0 280def error(msg): 281 global current_input_lineno 282 print("Error: %s%s" % ((("Line %d, " % current_input_lineno) 283 if current_input_lineno != 0 else ""), 284 msg)) 285 exit(1) 286 287if(test_name or description) and description_file: 288 error("both command-line testcase and input file specified") 289if test_name and not description: 290 error("command-line testcase name without description string") 291 292# Main class type describing a testcase. 293class TestDescr: 294 def __init__(self): 295 self.objs = [] # list of all DSO objects 296 self.deps = OrderedDict() # map of DSO object -> list of dependencies 297 298 # map of DSO object -> list of call refs 299 self.callrefs = OrderedDict() 300 301 # map of DSO object -> list of permutations of dependencies 302 self.dep_permutations = OrderedDict() 303 304 # map of DSO object -> SONAME of object (if one is specified) 305 self.soname_map = OrderedDict() 306 307 # list of main program operations 308 self.main_program = [] 309 # set if default dependencies added to main 310 self.main_program_default_deps = True 311 312 self.test_name = "" # name of testcase 313 self.expected_outputs = OrderedDict() # expected outputs of testcase 314 self.xfail = False # set if this is a XFAIL testcase 315 self.xtest = False # set if this is put under 'xtests' 316 317 # Add 'object -> [object, object, ...]' relations to CURR_MAP 318 def __add_deps_internal(self, src_objs, dst_objs, curr_map): 319 for src in src_objs: 320 for dst in dst_objs: 321 if not src in curr_map: 322 curr_map[src] = [] 323 if not dst in curr_map[src]: 324 curr_map[src].append(dst) 325 def add_deps(self, src_objs, dst_objs): 326 self.__add_deps_internal(src_objs, dst_objs, self.deps) 327 def add_callrefs(self, src_objs, dst_objs): 328 self.__add_deps_internal(src_objs, dst_objs, self.callrefs) 329 330# Process commands inside the {} construct. 331# Note that throughout this script, the main program object is represented 332# by the '#' string. 333def process_main_program(test_descr, mainprog_str): 334 if mainprog_str: 335 test_descr.main_program = mainprog_str.split(';') 336 for s in test_descr.main_program: 337 m = re.match(r"^([+\-%^@])([0-9a-zA-Z]+)$", s) 338 if not m: 339 error("'%s' is not recognized main program operation" % (s)) 340 opr = m.group(1) 341 obj = m.group(2) 342 if not obj in test_descr.objs: 343 test_descr.objs.append(obj) 344 if opr == '%' or opr == '@': 345 test_descr.add_callrefs(['#'], [obj]) 346 # We have a main program specified, turn this off 347 test_descr.main_program_default_deps = False 348 349# For(a1 a2 b1-12) object set descriptions, expand into an object list 350def expand_object_set_string(descr_str): 351 obj_list = [] 352 descr_list = descr_str.split() 353 for descr in descr_list: 354 m = re.match(r"^([a-zA-Z][0-9a-zA-Z]*)(-[0-9]+)?$", descr) 355 if not m: 356 error("'%s' is not a valid object set description" % (descr)) 357 obj = m.group(1) 358 idx_end = m.group(2) 359 if not idx_end: 360 if not obj in obj_list: 361 obj_list.append(obj) 362 else: 363 idx_end = int(idx_end[1:]) 364 m = re.match(r"^([0-9a-zA-Z][a-zA-Z]*)([0-9]+)$", obj) 365 if not m: 366 error("object description '%s' is malformed" % (obj)) 367 obj_name = m.group(1) 368 idx_start = int(m.group (2)) 369 if idx_start > idx_end: 370 error("index range %s-%s invalid" % (idx_start, idx_end)) 371 for i in range(idx_start, idx_end + 1): 372 o = obj_name + str(i) 373 if not o in obj_list: 374 obj_list.append(o) 375 return obj_list 376 377# Lexer for tokens 378tokenspec = [ ("SONAME", r"soname\(([0-9a-zA-Z{}]+)\)=([0-9a-zA-Z]+)"), 379 ("OBJ", r"([0-9a-zA-Z]+)"), 380 ("DEP", r"->"), 381 ("CALLREF", r"=>"), 382 ("OBJSET", r"\[([0-9a-zA-Z]+)\]"), 383 ("OBJSET2", r"\(([0-9a-zA-Z \-]+)\)"), 384 ("OBJSET3", r"\*"), 385 ("PROG", r"{([0-9a-zA-Z;+^\-%@]*)}"), 386 ("PERMUTE", r"!"), 387 ("SEMICOL", r";"), 388 ("ERROR", r".") ] 389tok_re = '|'.join('(?P<%s>%s)' % pair for pair in tokenspec) 390 391# Main line parser of description language 392def parse_description_string(t, descr_str): 393 # State used when parsing dependencies 394 curr_objs = [] 395 in_dep = False 396 in_callref = False 397 def clear_dep_state(): 398 nonlocal in_dep, in_callref 399 in_dep = in_callref = False 400 401 for m in re.finditer(tok_re, descr_str): 402 kind = m.lastgroup 403 value = m.group() 404 if kind == "SONAME": 405 s = re.match(r"soname\(([0-9a-zA-Z{}]+)\)=([0-9a-zA-Z]+)", value) 406 obj = s.group(1) 407 val = s.group(2) 408 if obj == "{}": 409 if '#' in t.soname_map: 410 error("soname of main program already set") 411 # Adjust to internal name 412 obj = '#' 413 else: 414 if re.match(r"[{}]", obj): 415 error("invalid object name '%s'" % (obj)) 416 if not obj in t.objs: 417 error("'%s' is not name of already defined object" % (obj)) 418 if obj in t.soname_map: 419 error("'%s' already has soname of '%s' set" 420 % (obj, t.soname_map[obj])) 421 t.soname_map[obj] = val 422 423 elif kind == "OBJ": 424 if in_dep: 425 t.add_deps(curr_objs, [value]) 426 elif in_callref: 427 t.add_callrefs(curr_objs, [value]) 428 clear_dep_state() 429 curr_objs = [value] 430 if not value in t.objs: 431 t.objs.append(value) 432 433 elif kind == "OBJSET": 434 objset = value[1:len(value)-1] 435 if in_dep: 436 t.add_deps(curr_objs, list (objset)) 437 elif in_callref: 438 t.add_callrefs(curr_objs, list (objset)) 439 clear_dep_state() 440 curr_objs = list(objset) 441 for o in list(objset): 442 if not o in t.objs: 443 t.objs.append(o) 444 445 elif kind == "OBJSET2": 446 descr_str = value[1:len(value)-1] 447 descr_str.strip() 448 objs = expand_object_set_string(descr_str) 449 if not objs: 450 error("empty object set '%s'" % (value)) 451 if in_dep: 452 t.add_deps(curr_objs, objs) 453 elif in_callref: 454 t.add_callrefs(curr_objs, objs) 455 clear_dep_state() 456 curr_objs = objs 457 for o in objs: 458 if not o in t.objs: 459 t.objs.append(o) 460 461 elif kind == "OBJSET3": 462 if in_dep: 463 t.add_deps(curr_objs, ['*']) 464 elif in_callref: 465 t.add_callrefs(curr_objs, ['*']) 466 else: 467 error("non-dependence target set '*' can only be used " 468 "as target of ->/=> operations") 469 clear_dep_state() 470 curr_objs = ['*'] 471 472 elif kind == "PERMUTE": 473 if in_dep or in_callref: 474 error("syntax error, permute operation invalid here") 475 if not curr_objs: 476 error("syntax error, no objects to permute here") 477 478 for obj in curr_objs: 479 if not obj in t.dep_permutations: 480 # Signal this object has permuted dependencies 481 t.dep_permutations[obj] = [] 482 483 elif kind == "PROG": 484 if t.main_program: 485 error("cannot have more than one main program") 486 if in_dep: 487 error("objects cannot have dependency on main program") 488 if in_callref: 489 # TODO: A DSO can resolve to a symbol in the main binary, 490 # which we syntactically allow here, but haven't yet 491 # implemented. 492 t.add_callrefs(curr_objs, ["#"]) 493 process_main_program(t, value[1:len(value)-1]) 494 clear_dep_state() 495 curr_objs = ["#"] 496 497 elif kind == "DEP": 498 if in_dep or in_callref: 499 error("syntax error, multiple contiguous ->,=> operations") 500 if '*' in curr_objs: 501 error("non-dependence target set '*' can only be used " 502 "as target of ->/=> operations") 503 in_dep = True 504 505 elif kind == "CALLREF": 506 if in_dep or in_callref: 507 error("syntax error, multiple contiguous ->,=> operations") 508 if '*' in curr_objs: 509 error("non-dependence target set '*' can only be used " 510 "as target of ->/=> operations") 511 in_callref = True 512 513 elif kind == "SEMICOL": 514 curr_objs = [] 515 clear_dep_state() 516 517 else: 518 error("unknown token '%s'" % (value)) 519 return t 520 521# Main routine to process each testcase description 522def process_testcase(t): 523 global objpfx 524 assert t.test_name 525 526 base_test_name = t.test_name 527 test_subdir = base_test_name + "-dir" 528 testpfx = objpfx + test_subdir + "/" 529 test_srcdir = "dso-sort-tests-src/" 530 testpfx_src = objpfx + test_srcdir 531 532 if not os.path.exists(testpfx): 533 os.mkdir(testpfx) 534 if not os.path.exists(testpfx_src): 535 os.mkdir(testpfx_src) 536 537 def find_objs_not_depended_on(t): 538 objs_not_depended_on = [] 539 for obj in t.objs: 540 skip = False 541 for r in t.deps.items(): 542 if obj in r[1]: 543 skip = True 544 break 545 if not skip: 546 objs_not_depended_on.append(obj) 547 return objs_not_depended_on 548 549 non_dep_tgt_objs = find_objs_not_depended_on(t) 550 for obj in t.objs: 551 if obj in t.deps: 552 deps = t.deps[obj] 553 if '*' in deps: 554 deps.remove('*') 555 t.add_deps([obj], non_dep_tgt_objs) 556 if obj in t.callrefs: 557 deps = t.callrefs[obj] 558 if '*' in deps: 559 deps.remove('*') 560 t.add_callrefs([obj], non_dep_tgt_objs) 561 if "#" in t.deps: 562 deps = t.deps["#"] 563 if '*' in deps: 564 deps.remove('*') 565 t.add_deps(["#"], non_dep_tgt_objs) 566 567 # If no main program was specified in dependency description, make a 568 # default main program with deps pointing to all DSOs which are not 569 # depended by another DSO. 570 if t.main_program_default_deps: 571 main_deps = non_dep_tgt_objs 572 if not main_deps: 573 error("no objects for default main program to point " 574 "dependency to(all objects strongly connected?)") 575 t.add_deps(["#"], main_deps) 576 577 # Some debug output 578 if cmdlineargs.debug_output: 579 print("Testcase: %s" % (t.test_name)) 580 print("All objects: %s" % (t.objs)) 581 print("--- Static link dependencies ---") 582 for r in t.deps.items(): 583 print("%s -> %s" % (r[0], r[1])) 584 print("--- Objects whose dependencies are to be permuted ---") 585 for r in t.dep_permutations.items(): 586 print("%s" % (r[0])) 587 print("--- Call reference dependencies ---") 588 for r in t.callrefs.items(): 589 print("%s => %s" % (r[0], r[1])) 590 print("--- main program ---") 591 print(t.main_program) 592 593 # Main testcase generation routine, does Makefile fragment generation, 594 # testcase source generation, and if --build specified builds testcase. 595 def generate_testcase(test_descr, test_suffix): 596 597 test_name = test_descr.test_name + test_suffix 598 599 # Print out needed Makefile fragments for use in glibc/elf/Makefile. 600 module_names = "" 601 for o in test_descr.objs: 602 rule = ("$(objpfx)" + test_subdir + "/" + test_name 603 + "-" + o + ".os: $(objpfx)" + test_srcdir 604 + test_name + "-" + o + ".c\n" 605 "\t$(compile.c) $(OUTPUT_OPTION)\n") 606 makefile.write (rule) 607 module_names += " " + test_subdir + "/" + test_name + "-" + o 608 makefile.write("modules-names +=%s\n" % (module_names)) 609 610 # Depth-first traversal, executing FN(OBJ) in post-order 611 def dfs(t, fn): 612 def dfs_rec(obj, fn, obj_visited): 613 if obj in obj_visited: 614 return 615 obj_visited[obj] = True 616 if obj in t.deps: 617 for dep in t.deps[obj]: 618 dfs_rec(dep, fn, obj_visited) 619 fn(obj) 620 621 obj_visited = {} 622 for obj in t.objs: 623 dfs_rec(obj, fn, obj_visited) 624 625 # Generate link dependencies for all DSOs, done in a DFS fashion. 626 # Usually this doesn't need to be this complex, just listing the direct 627 # dependencies is enough. However to support creating circular 628 # dependency situations, traversing it by DFS and tracking processing 629 # status is the natural way to do it. 630 obj_processed = {} 631 fake_created = {} 632 def gen_link_deps(obj): 633 if obj in test_descr.deps: 634 dso = test_subdir + "/" + test_name + "-" + obj + ".so" 635 dependencies = "" 636 for dep in test_descr.deps[obj]: 637 if dep in obj_processed: 638 depstr = (" $(objpfx)" + test_subdir + "/" 639 + test_name + "-" + dep + ".so") 640 else: 641 # A circular dependency is satisfied by making a 642 # fake DSO tagged with the correct SONAME 643 depstr = (" $(objpfx)" + test_subdir + "/" 644 + test_name + "-" + dep + ".FAKE.so") 645 # Create empty C file and Makefile fragments for fake 646 # object. This only needs to be done at most once for 647 # an object name. 648 if not dep in fake_created: 649 f = open(testpfx_src + test_name + "-" + dep 650 + ".FAKE.c", "w") 651 f.write(" \n") 652 f.close() 653 # Generate rule to create fake object 654 makefile.write \ 655 ("LDFLAGS-%s = -Wl,--no-as-needed " 656 "-Wl,-soname=%s\n" 657 % (test_name + "-" + dep + ".FAKE.so", 658 ("$(objpfx)" + test_subdir + "/" 659 + test_name + "-" + dep + ".so"))) 660 rule = ("$(objpfx)" + test_subdir + "/" 661 + test_name + "-" + dep + ".FAKE.os: " 662 "$(objpfx)" + test_srcdir 663 + test_name + "-" + dep + ".FAKE.c\n" 664 "\t$(compile.c) $(OUTPUT_OPTION)\n") 665 makefile.write (rule) 666 makefile.write \ 667 ("modules-names += %s\n" 668 % (test_subdir + "/" 669 + test_name + "-" + dep + ".FAKE")) 670 fake_created[dep] = True 671 dependencies += depstr 672 makefile.write("$(objpfx)%s:%s\n" % (dso, dependencies)) 673 # Mark obj as processed 674 obj_processed[obj] = True 675 676 dfs(test_descr, gen_link_deps) 677 678 # Print LDFLAGS-* and *-no-z-defs 679 for o in test_descr.objs: 680 dso = test_name + "-" + o + ".so" 681 ldflags = "-Wl,--no-as-needed" 682 if o in test_descr.soname_map: 683 soname = ("$(objpfx)" + test_subdir + "/" 684 + test_name + "-" 685 + test_descr.soname_map[o] + ".so") 686 ldflags += (" -Wl,-soname=" + soname) 687 makefile.write("LDFLAGS-%s = %s\n" % (dso, ldflags)) 688 if o in test_descr.callrefs: 689 makefile.write("%s-no-z-defs = yes\n" % (dso)) 690 691 # Print dependencies for main test program. 692 depstr = "" 693 if '#' in test_descr.deps: 694 for o in test_descr.deps['#']: 695 depstr += (" $(objpfx)" + test_subdir + "/" 696 + test_name + "-" + o + ".so") 697 makefile.write("$(objpfx)%s/%s:%s\n" % (test_subdir, test_name, depstr)) 698 ldflags = "-Wl,--no-as-needed" 699 if '#' in test_descr.soname_map: 700 soname = ("$(objpfx)" + test_subdir + "/" 701 + test_name + "-" 702 + test_descr.soname_map['#'] + ".so") 703 ldflags += (" -Wl,-soname=" + soname) 704 makefile.write("LDFLAGS-%s = %s\n" % (test_name, ldflags)) 705 rule = ("$(objpfx)" + test_subdir + "/" + test_name + ".o: " 706 "$(objpfx)" + test_srcdir + test_name + ".c\n" 707 "\t$(compile.c) $(OUTPUT_OPTION)\n") 708 makefile.write (rule) 709 710 not_depended_objs = find_objs_not_depended_on(test_descr) 711 if not_depended_objs: 712 depstr = "" 713 for dep in not_depended_objs: 714 depstr += (" $(objpfx)" + test_subdir + "/" 715 + test_name + "-" + dep + ".so") 716 makefile.write("$(objpfx)%s.out:%s\n" % (base_test_name, depstr)) 717 718 # Add main executable to test-srcs 719 makefile.write("test-srcs += %s/%s\n" % (test_subdir, test_name)) 720 # Add dependency on main executable of test 721 makefile.write("$(objpfx)%s.out: $(objpfx)%s/%s\n" 722 % (base_test_name, test_subdir, test_name)) 723 724 for r in test_descr.expected_outputs.items(): 725 tunable_options = [] 726 specific_tunable = r[0] 727 xfail = r[1][1] 728 if specific_tunable != "": 729 tunable_options = [specific_tunable] 730 else: 731 tunable_options = default_tunable_options 732 if not tunable_options: 733 tunable_options = [""] 734 735 for tunable in tunable_options: 736 tunable_env = "" 737 tunable_sfx = "" 738 exp_tunable_sfx = "" 739 if tunable: 740 tunable_env = "GLIBC_TUNABLES=%s " % tunable 741 tunable_sfx = "-" + tunable.replace("=","_") 742 if specific_tunable: 743 tunable_sfx = "-" + specific_tunable.replace("=","_") 744 exp_tunable_sfx = tunable_sfx 745 tunable_descr = ("(%s)" % tunable_env.strip() 746 if tunable_env else "") 747 # Write out fragment of shell script for this single test. 748 test_descr.sh.write \ 749 ("%s${test_wrapper_env} ${run_program_env} \\\n" 750 "${common_objpfx}support/test-run-command \\\n" 751 "${common_objpfx}elf/ld.so \\\n" 752 "--library-path ${common_objpfx}elf/%s:" 753 "${common_objpfx}elf:${common_objpfx}.:" 754 "${common_objpfx}dlfcn \\\n" 755 "${common_objpfx}elf/%s/%s > \\\n" 756 " ${common_objpfx}elf/%s/%s%s.output\n" 757 % (tunable_env ,test_subdir, 758 test_subdir, test_name, test_subdir, test_name, 759 tunable_sfx)) 760 # Generate a run of each test and compare with expected out 761 test_descr.sh.write \ 762 ("if [ $? -ne 0 ]; then\n" 763 " echo '%sFAIL: %s%s execution test'\n" 764 " something_failed=true\n" 765 "else\n" 766 " diff -wu ${common_objpfx}elf/%s/%s%s.output \\\n" 767 " ${common_objpfx}elf/%s%s%s.exp\n" 768 " if [ $? -ne 0 ]; then\n" 769 " echo '%sFAIL: %s%s expected output comparison'\n" 770 " something_failed=true\n" 771 " fi\n" 772 "fi\n" 773 % (("X" if xfail else ""), test_name, tunable_descr, 774 test_subdir, test_name, tunable_sfx, 775 test_srcdir, base_test_name, exp_tunable_sfx, 776 ("X" if xfail else ""), test_name, tunable_descr)) 777 778 # Generate C files according to dependency and calling relations from 779 # description string. 780 for obj in test_descr.objs: 781 src_name = test_name + "-" + obj + ".c" 782 f = open(testpfx_src + src_name, "w") 783 if obj in test_descr.callrefs: 784 called_objs = test_descr.callrefs[obj] 785 for callee in called_objs: 786 f.write("extern void fn_%s (void);\n" % (callee)) 787 if len(obj) == 1: 788 f.write("extern int putchar(int);\n") 789 f.write("static void __attribute__((constructor)) " + 790 "init(void){putchar('%s');putchar('>');}\n" % (obj)) 791 f.write("static void __attribute__((destructor)) " + 792 "fini(void){putchar('<');putchar('%s');}\n" % (obj)) 793 else: 794 f.write('extern int printf(const char *, ...);\n') 795 f.write('static void __attribute__((constructor)) ' + 796 'init(void){printf("%s>");}\n' % (obj)) 797 f.write('static void __attribute__((destructor)) ' + 798 'fini(void){printf("<%s");}\n' % (obj)) 799 if obj in test_descr.callrefs: 800 called_objs = test_descr.callrefs[obj] 801 if len(obj) != 1: 802 f.write("extern int putchar(int);\n") 803 f.write("void fn_%s (void) {\n" % (obj)) 804 if len(obj) == 1: 805 f.write(" putchar ('%s');\n" % (obj)); 806 f.write(" putchar ('(');\n"); 807 else: 808 f.write(' printf ("%s(");\n' % (obj)); 809 for callee in called_objs: 810 f.write(" fn_%s ();\n" % (callee)) 811 f.write(" putchar (')');\n"); 812 f.write("}\n") 813 else: 814 for callref in test_descr.callrefs.items(): 815 if obj in callref[1]: 816 if len(obj) == 1: 817 # We need to declare printf here in this case. 818 f.write('extern int printf(const char *, ...);\n') 819 f.write("void fn_%s (void) {\n" % (obj)) 820 f.write(' printf ("%s()");\n' % (obj)) 821 f.write("}\n") 822 break 823 f.close() 824 825 # Open C file for writing main program 826 f = open(testpfx_src + test_name + ".c", "w") 827 828 # if there are some operations in main(), it means we need -ldl 829 f.write("#include <stdio.h>\n") 830 f.write("#include <stdlib.h>\n") 831 f.write("#include <dlfcn.h>\n") 832 for s in test_descr.main_program: 833 if s[0] == '@': 834 f.write("extern void fn_%s (void);\n" % (s[1:])); 835 f.write("int main (void) {\n") 836 f.write(" putchar('{');\n") 837 838 # Helper routine for generating sanity checking code. 839 def put_fail_check(fail_cond, action_desc): 840 f.write(' if (%s) { printf ("\\n%s failed: %%s\\n", ' 841 'dlerror()); exit (1);}\n' % (fail_cond, action_desc)) 842 i = 0 843 while i < len(test_descr.main_program): 844 s = test_descr.main_program[i] 845 obj = s[1:] 846 dso = test_name + "-" + obj 847 if s[0] == '+' or s[0] == '^': 848 if s[0] == '+': 849 dlopen_flags = "RTLD_LAZY|RTLD_GLOBAL" 850 f.write(" putchar('+');\n"); 851 else: 852 dlopen_flags = "RTLD_LAZY" 853 f.write(" putchar(':');\n"); 854 if len(obj) == 1: 855 f.write(" putchar('%s');\n" % (obj)); 856 else: 857 f.write(' printf("%s");\n' % (obj)); 858 f.write(" putchar('[');\n"); 859 f.write(' void *%s = dlopen ("%s.so", %s);\n' 860 % (obj, dso, dlopen_flags)) 861 put_fail_check("!%s" % (obj), 862 "%s.so dlopen" % (dso)) 863 f.write(" putchar(']');\n"); 864 elif s[0] == '-': 865 f.write(" putchar('-');\n"); 866 if len(obj) == 1: 867 f.write(" putchar('%s');\n" % (obj)); 868 else: 869 f.write(' printf("%s");\n' % (obj)); 870 f.write(" putchar('[');\n"); 871 put_fail_check("dlclose (%s) != 0" % (obj), 872 "%s.so dlclose" % (dso)) 873 f.write(" putchar(']');\n"); 874 elif s[0] == '%': 875 f.write(" putchar('%');\n"); 876 f.write(' void (*fn_%s)(void) = dlsym (%s, "fn_%s");\n' 877 % (obj, obj, obj)) 878 put_fail_check("!fn_%s" % (obj), 879 "dlsym(fn_%s) from %s.so" % (obj, dso)) 880 f.write(" fn_%s ();\n" % (obj)) 881 elif s[0] == '@': 882 f.write(" putchar('@');\n"); 883 f.write(" fn_%s ();\n" % (obj)) 884 f.write(" putchar(';');\n"); 885 i += 1 886 f.write(" putchar('}');\n") 887 f.write(" return 0;\n") 888 f.write("}\n") 889 f.close() 890 891 # --build option processing: build generated sources using 'build_gcc' 892 if cmdlineargs.build: 893 # Helper routine to run a shell command, for running GCC below 894 def run_cmd(args): 895 cmd = str.join(' ', args) 896 if cmdlineargs.debug_output: 897 print(cmd) 898 p = subprocess.Popen(args) 899 p.wait() 900 if p.returncode != 0: 901 error("error running command: %s" % (cmd)) 902 903 # Compile individual .os files 904 for obj in test_descr.objs: 905 src_name = test_name + "-" + obj + ".c" 906 obj_name = test_name + "-" + obj + ".os" 907 run_cmd([build_gcc, "-c", "-fPIC", testpfx_src + src_name, 908 "-o", testpfx + obj_name]) 909 910 obj_processed = {} 911 fake_created = {} 912 # Function to create <test_name>-<obj>.so 913 def build_dso(obj): 914 obj_name = test_name + "-" + obj + ".os" 915 dso_name = test_name + "-" + obj + ".so" 916 deps = [] 917 if obj in test_descr.deps: 918 for dep in test_descr.deps[obj]: 919 if dep in obj_processed: 920 deps.append(dep) 921 else: 922 deps.append(dep + ".FAKE") 923 if not dep in fake_created: 924 base_name = testpfx + test_name + "-" + dep 925 src_base_name = (testpfx_src + test_name 926 + "-" + dep) 927 cmd = [build_gcc, "-Wl,--no-as-needed", 928 ("-Wl,-soname=" + base_name + ".so"), 929 "-shared", base_name + ".FAKE.c", 930 "-o", src_base_name + ".FAKE.so"] 931 run_cmd(cmd) 932 fake_created[dep] = True 933 dso_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", 934 deps) 935 cmd = [build_gcc, "-shared", "-o", testpfx + dso_name, 936 testpfx + obj_name, "-Wl,--no-as-needed"] 937 if obj in test_descr.soname_map: 938 soname = ("-Wl,-soname=" + testpfx + test_name + "-" 939 + test_descr.soname_map[obj] + ".so") 940 cmd += [soname] 941 cmd += list(dso_deps) 942 run_cmd(cmd) 943 obj_processed[obj] = True 944 945 # Build all DSOs, this needs to be in topological dependency order, 946 # or link will fail 947 dfs(test_descr, build_dso) 948 949 # Build main program 950 deps = [] 951 if '#' in test_descr.deps: 952 deps = test_descr.deps['#'] 953 main_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", 954 deps) 955 cmd = [build_gcc, "-Wl,--no-as-needed", "-o", testpfx + test_name, 956 testpfx_src + test_name + ".c", "-L%s" % (os.getcwd()), 957 "-Wl,-rpath-link=%s" % (os.getcwd())] 958 if '#' in test_descr.soname_map: 959 soname = ("-Wl,-soname=" + testpfx + test_name + "-" 960 + test_descr.soname_map['#'] + ".so") 961 cmd += [soname] 962 cmd += list(main_deps) 963 run_cmd(cmd) 964 965 # Check if we need to enumerate permutations of dependencies 966 need_permutation_processing = False 967 if t.dep_permutations: 968 # Adjust dep_permutations into map of object -> dependency permutations 969 for r in t.dep_permutations.items(): 970 obj = r[0] 971 if obj in t.deps and len(t.deps[obj]) > 1: 972 deps = t.deps[obj] 973 t.dep_permutations[obj] = list(itertools.permutations (deps)) 974 need_permutation_processing = True 975 976 def enum_permutations(t, perm_list): 977 test_subindex = 1 978 curr_perms = [] 979 def enum_permutations_rec(t, perm_list): 980 nonlocal test_subindex, curr_perms 981 if len(perm_list) >= 1: 982 curr = perm_list[0] 983 obj = curr[0] 984 perms = curr[1] 985 if not perms: 986 # This may be an empty list if no multiple dependencies to 987 # permute were found, skip to next in this case 988 enum_permutations_rec(t, perm_list[1:]) 989 else: 990 for deps in perms: 991 t.deps[obj] = deps 992 permstr = "" if obj == "#" else obj + "_" 993 permstr += str.join('', deps) 994 curr_perms.append(permstr) 995 enum_permutations_rec(t, perm_list[1:]) 996 curr_perms = curr_perms[0:len(curr_perms)-1] 997 else: 998 # t.deps is now instantiated with one dependency order 999 # permutation(across all objects that have multiple 1000 # permutations), now process a testcase 1001 generate_testcase(t, ("_" + str (test_subindex) 1002 + "-" + str.join('-', curr_perms))) 1003 test_subindex += 1 1004 enum_permutations_rec(t, perm_list) 1005 1006 # Create *.exp files with expected outputs 1007 for r in t.expected_outputs.items(): 1008 sfx = "" 1009 if r[0] != "": 1010 sfx = "-" + r[0].replace("=","_") 1011 f = open(testpfx_src + t.test_name + sfx + ".exp", "w") 1012 (output, xfail) = r[1] 1013 f.write('%s' % output) 1014 f.close() 1015 1016 # Create header part of top-level testcase shell script, to wrap execution 1017 # and output comparison together. 1018 t.sh = open(testpfx_src + t.test_name + ".sh", "w") 1019 t.sh.write("#!/bin/sh\n") 1020 t.sh.write("# Test driver for %s, generated by " 1021 "dso-ordering-test.py\n" % (t.test_name)) 1022 t.sh.write("common_objpfx=$1\n") 1023 t.sh.write("test_wrapper_env=$2\n") 1024 t.sh.write("run_program_env=$3\n") 1025 t.sh.write("something_failed=false\n") 1026 1027 # Starting part of Makefile fragment 1028 makefile.write("ifeq (yes,$(build-shared))\n") 1029 1030 if need_permutation_processing: 1031 enum_permutations(t, list (t.dep_permutations.items())) 1032 else: 1033 # We have no permutations to enumerate, just process testcase normally 1034 generate_testcase(t, "") 1035 1036 # If testcase is XFAIL, indicate so 1037 if t.xfail: 1038 makefile.write("test-xfail-%s = yes\n" % t.test_name) 1039 1040 # Output end part of Makefile fragment 1041 expected_output_files = "" 1042 for r in t.expected_outputs.items(): 1043 sfx = "" 1044 if r[0] != "": 1045 sfx = "-" + r[0].replace("=","_") 1046 expected_output_files += " $(objpfx)%s%s%s.exp" % (test_srcdir, 1047 t.test_name, sfx) 1048 makefile.write \ 1049 ("$(objpfx)%s.out: $(objpfx)%s%s.sh%s " 1050 "$(common-objpfx)support/test-run-command\n" 1051 % (t.test_name, test_srcdir, t.test_name, 1052 expected_output_files)) 1053 makefile.write("\t$(SHELL) $< $(common-objpfx) '$(test-wrapper-env)' " 1054 "'$(run-program-env)' > $@; $(evaluate-test)\n") 1055 makefile.write("ifeq ($(run-built-tests),yes)\n") 1056 if t.xtest: 1057 makefile.write("xtests-special += $(objpfx)%s.out\n" % (t.test_name)) 1058 else: 1059 makefile.write("tests-special += $(objpfx)%s.out\n" % (t.test_name)) 1060 makefile.write("endif\n") 1061 makefile.write("endif\n") 1062 1063 # Write ending part of shell script generation 1064 t.sh.write("if $something_failed; then\n" 1065 " exit 1\n" 1066 "else\n" 1067 " echo '%sPASS: all tests for %s succeeded'\n" 1068 " exit 0\n" 1069 "fi\n" % (("X" if t.xfail else ""), 1070 t.test_name)) 1071 t.sh.close() 1072 1073# Decription file parsing 1074def parse_description_file(filename): 1075 global default_tunable_options 1076 global current_input_lineno 1077 f = open(filename) 1078 if not f: 1079 error("cannot open description file %s" % (filename)) 1080 descrfile_lines = f.readlines() 1081 t = None 1082 for line in descrfile_lines: 1083 p = re.compile(r"#.*$") 1084 line = p.sub("", line) # Filter out comments 1085 line = line.strip() # Remove excess whitespace 1086 current_input_lineno += 1 1087 1088 m = re.match(r"^tunable_option:\s*(.*)$", line) 1089 if m: 1090 if m.group(1) == "": 1091 error("tunable option cannot be empty") 1092 default_tunable_options.append(m.group (1)) 1093 continue 1094 1095 m = re.match(r"^clear_tunables$", line) 1096 if m: 1097 default_tunable_options = [] 1098 continue 1099 1100 m = re.match(r"^([^:]+):\s*(.*)$", line) 1101 if m: 1102 lhs = m.group(1) 1103 o = re.match(r"^output(.*)$", lhs) 1104 xfail = False 1105 if not o: 1106 o = re.match(r"^xfail_output(.*)$", lhs) 1107 if o: 1108 xfail = True; 1109 if o: 1110 if not t: 1111 error("output specification without testcase description") 1112 tsstr = "" 1113 if o.group(1): 1114 ts = re.match(r"^\(([a-zA-Z0-9_.=]*)\)$", o.group (1)) 1115 if not ts: 1116 error("tunable option malformed '%s'" % o.group(1)) 1117 tsstr = ts.group(1) 1118 t.expected_outputs[tsstr] = (m.group(2), xfail) 1119 # Any tunable option XFAILed means entire testcase 1120 # is XFAIL/XPASS 1121 t.xfail |= xfail 1122 else: 1123 if t: 1124 # Starting a new test description, end and process 1125 # current one. 1126 process_testcase(t) 1127 t = TestDescr() 1128 x = re.match(r"^xtest\((.*)\)$", lhs) 1129 if x: 1130 t.xtest = True 1131 t.test_name = x.group(1) 1132 else: 1133 t.test_name = lhs 1134 descr_string = m.group(2) 1135 parse_description_string(t, descr_string) 1136 continue 1137 else: 1138 if line: 1139 if not t: 1140 error("no active testcase description") 1141 parse_description_string(t, line) 1142 # Process last completed test description 1143 if t: 1144 process_testcase(t) 1145 1146# Setup Makefile output to file or stdout as selected 1147if output_makefile: 1148 output_makefile_dir = os.path.dirname(output_makefile) 1149 if output_makefile_dir: 1150 os.makedirs(output_makefile_dir, exist_ok = True) 1151 makefile = open(output_makefile, "w") 1152else: 1153 makefile = open(sys.stdout.fileno (), "w") 1154 1155# Finally, the main top-level calling of above parsing routines. 1156if description_file: 1157 parse_description_file(description_file) 1158else: 1159 t = TestDescr() 1160 t.test_name = test_name 1161 parse_description_string(t, description) 1162 process_testcase(t) 1163 1164# Close Makefile fragment output 1165makefile.close() 1166