Artifact for OptFuzz

OPTFuzz: Artifact and Benchmark for Compiler Testing This artifact accompanies our project OPTFuzz, a compiler testing framework that leverages large language models (LLMs) to generate fine-grained compilation options for exploring complex compilation spaces. OPTFuzz uses LLMs in two main stages. First, it extracts bug-triggering code snippets from historical compiler bug reports and uses them as test programs. Second, it generates compilation-option sequences tailored to the input program, in order to trigger compiler defects more effectively. Project Summary: Background: Compilers are critical infrastructure in modern software development, and their quality directly affects the reliability and performance of downstream software systems. Although compiler testing has been extensively studied, existing approaches often focus on restricted subsets of compilation options and do not sufficiently explore more complex option combinations. Challenges: Manually designing effective combinations of compiler options is labor-intensive and requires substantial expertise, due to the large number of options and their complicated interactions. Approach: The OPTFUZZ framework enhances compiler testing through the following steps: Program Acquisition: Extracts test programs from historical GCC and LLVM bug reports. Code Abstraction: Abstracts source code using an intermediate-representation (IR)-based method to preserve key structural information. Compilation Option Generation: Uses LLMs to generate complex compilation-option sequences that may expose compiler defects. Test Result Processing: Executes compiler tests and analyzes outcomes such as internal compiler errors (ICEs) and hang issues. Contributions: The main contributions of OPTFUZZ include a novel approach to using LLMs to obtain code snippets from historical issue reports, abstracting source code at the IR level, and combining LLMs to generate complex compilation options for compiler testing. Benchmark: We systematically mined and curated historical bug reports from both GCC and LLVM to build a benchmark dataset. This dataset contains a diverse collection of historical bug-triggering programs and is released to support further research in compiler testing. The artifact includes a compressed archive, src.tar.xz, containing all crawled bug reports and extracted code snippets. In total, we collected approximately 110,939 test programs from 84,555 bug reports. The provided scripts can also be used to crawl and process bug reports independently. Experimental Results: Through extensive experiments on GCC and LLVM, OPTFUZZ has demonstrated superior bug detection capabilities compared to random compilation space exploration methods and other existing technologies. Practical Utility: OPTFUZZ identified 64 bugs in GCC and LLVM, with 53 confirmed or fixed, highlighting the practical utility of the method. Note: The script requires an internet connection to get data from Bugzilla and Github. The script requires a Python environment and the installation of listed libraries. Dependency libraries: github: Used to access the GitHub API. urlextract: Used to extract URLs from text. mistune: Used to parse Markdown text. requests: Used for HTTP requests. beautifulsoup4: Used to paser HTML file. shlex: Used to split strings into shell-like syntax. tqdm: Used to add a progress bar to loops and other iterables, providing a visual indication of progress. colorama: Used to emit colored output in terminals across different operating systems. signal: Used to provide mechanisms to use signal handlers for asynchronous events. from google import genai: Used to Call Gemini API(Could be replaced to other LLMs) File structure: artifact: | |--bugreport--|--gcc---|--GCC_report_get.py | | |-----llm_for_code.py | | | |--llvm--|--LLVM_report_get.py | |--llm_for_code.py | |--irplugin--|--generate_ir.py | |--makefile | |--plugin.cpp | |--optionstest--|--gcc--|--gcc_option_test.py | | | |--llvm-|--llvm_option_test.py src.tar.xz: A compressed archive containing all crawled bug reports and their code snippets. We obtained around 110,939 test programs from 84,555 bug reports. Of course, you can also use the provided scripts to collect programs for testing yourself. Bugs Found by OPTFuzz: GCC bug-115411 : ICE : in expand_call, at calls.cc:3668 GCC bug-115412: ICE: canonical types differ for identical types ‘stdis_sametypename fooTtype, U’ and ‘stdis_sameT, U’ GCC bug-115426: ICE: in execute_todo, at passes.cc:2138 GCC bug-115431: ICE: tree check: expected tree that contains ‘decl common’ structure, have ‘error_mark’ in decl_template_parm_check, at cp/cp-tree.h:5131 GCC bug-115469: [14 Regression] ICE :tree check expected class 'type', have 'exceptional' (error_mark) in poplevel_named_label_1, at cpdecl.cc579 GCC bug-115489: [12/13/14/15 regression] ICE: tree check: expected class 'type', have 'exceptional' (error_mark) in create_tmp_from_val, at gimplify.cc:589 since r12-3278-g823685221de986 GCC bug-115501: [13/14 Regression] ICE: in build_call_a with dynamic_cast after invalid definition of __cxxabiv1::__dynamic_cast since r13-3299 GCC bug-115510: ICE : Segmentation fault in build_new_method_call and finish_call_expr GCC bug-115560: ICE: in type_dependent_expression_p, at cp/pt.cc:28576 GCC bug-115572: ICE: in dependent_type_p, at cp/pt.cc:28020 GCC bug-115588: ICE: in tsubst_stmt, at cp/pt.cc:18527 GCC bug-115599: ICE: qsort checking failed during GIMPLE pass: reassoc (error: qsort comparator non-negative on sorted output: 150142972) GCC bug-115620: ICE: in tsubst_pack_expansion, at cp/pt.cc:13703 GCC bug-115623: ICE: Segmentation fault in finish_for_cond with novector and almost infinite loop GCC bug-115786: ICE: Segmentation fault (add_stmt at ./gcc/gcc/c/c-decl.cc:689 and c_parser_declaration_or_fndef at ./gcc/gcc/c/c-parser.cc:3027) GCC bug-115787: [GIMPLE-FE] ICE: in gimple_build_switch_nlabels, at gimple.cc:807 GCC bug-115919: ICE: in tsubst_expr, at cp/pt.cc:20300 GCC bug-115930: ICE: tree check: expected type_argument_pack or nontype_argument_pack, have integer_type in template_parm_natural_p, at cp/mangle.cc:1828 GCC bug-115940: ICE: tree check: expected record_type or union_type or qual_union_type, have translation_unit_decl in maybe_dummy_object, at cp/tree.cc:4379 GCC bug-116002: GCC Compiler time-hog with large basic block in Function GCC bug-116042: ICE Segmentation fault ( in ix86_finalize_stack_frame_flags and ix86_expand_prologue()) GCC bug-116113: [15 Regression] ICE: Segmentation fault (maybe_convert_cond) GCC bug-116320: [12/13 Regression] ICE: Segmentation fault (perform_or_defer_access_check) since r11-1350 GCC bug-116323: [12/13/14/15 Regression] ICE: tree check: expected record_type or union_type or qual_union_type, have bound_template_template_parm in access_in_type, at cp/search.cc:663 GCC bug-117065: ICE: tree check: expected class 'type', have 'exceptional' (error_mark) in type_has_padding_at_level_p, at gimple-fold.cc:4820 GCC bug-117083: ICE: in get_expr_operands, at tree-ssa-operands.cc:939 GCC bug-117091: switch clustering takes extensive time with large switches even at -O0 GCC bug-117455: ICE: in gimple_build_assign_1, at gimple.cc:480 GCC bug-117652: [14/15 regression] ICE: tree check: expected class ‘type’, have ‘exceptional’ (error_mark) in tagged_types_tu_compatible_p, at c/c-typeck.cc:1919 GCC bug-117806: [14/15 Regression] ICE: in composite_type_internal with C23 tag compatibility and packed/aligned attribute on a field (defined 3 times) GCC bug-117826: [15 Regression] ICE: tree check: expected tree that contains 'decl minimal' structure, have 'tree_list' in hash, at tree.h:5958 GCC bug-117837: ICE: verify_cgraph_node failed GCC bug-117845: [14/15 Regression] ICE in pass eh after error with -fsanitize=address GCC bug-117856: - -aux-info not expecting error_mark for types of arguments GCC bug-117898: ICE: tree check: accessed elt 1 of 'tree_vec' with 0 elts in make_pack_index, at cp/pt.cc:4290 GCC bug-118290: [14/15 Regression] ICE: argc.1 from nested referenced in nested GCC bug-119842: [15/16 Regression] ICE: in c_type_tag, at c/c-typeck.cc:606 GCC bug-120557: [14/15/16 Regression] ICE: tree check: expected record_type or union_type or qual_union_type, have integer_type in finish_non_static_data_member, at cp/semantics.cc:2809 GCC bug-120558: ICE: Segmentation fault at munge with ` -MMD -MQ ./ ` GCC bug-120563: ICE: tree check: expected class ‘type’, have ‘exceptional’ (error_mark) in c_parser_omp_clause_reduction, at c/c-parser.cc:18549 GCC bug-120564: [13/14 Regression] ICE: verify_gimple failed during IPA pass: *free_lang_data since r8-3241 GCC bug-120575: [15/16 Regression] ICE: in cp_parser_abort_tentative_parse, at cp/parser.cc:36574 GCC bug-120602: ICE: in pp_string, at pretty-print.cc:2652 GCC bug-120618: ICE: tree check: expected template_type_parm, have boolean_type in write_requirement, at cp/mangle.cc:3134 LLVM bug-95366: Clang-19 crash:/lib/AST/ExprConstant.cpp:1633: void {anonymous}::LValue::setFrom(clang::ASTContext&, const clang::APValue&): Assertion `V.isLValue() && "Setting LValue from a non-LValue?"' failed. LLVM bug-95420: Clang-19 crash: void* clang::DeclarationName::getFETokenInfo() const: Assertion `getPtr() && "getFETokenInfo on an empty DeclarationName!"' failed. LLVM bug-95495: Clang-19 crash: const clang::FieldDecl* clang::APValue::getUnionField() const: Assertion `isUnion() && "Invalid accessor"' failed. LLVM bug-95500: Clang-19 crash: fatal error: error in backend: register rbp is allocatable: function has no frame pointer LLVM bug-97780: Clang-19 cras