Structural Mutation Based Differential Testing for FPGA Logic Synthesis Compilers

Field Programmable Gate Arrays (FPGAs) play a crucial role in Electronic Design Automation (EDA) applications, which have been widely used in safety-critical environments, including aerospace, chip manufacturing, and medical devices. A critical stage in FPGA development is logic synthesis, which enables developers to translate their software designs into hardware net lists. However, bugs in FPGA logic synthesis compilers may lead to unexpected behaviors in target hardware net lists, which may cause potential security risks. Therefore, it is critical to detect and repair such bugs in FPGA logic synthesis compilers. While recent testing methods have made progress in automatic bug detection, they remain limited by a fundamental weakness, the generated test cases exhibit minimal timing complexity and structural diversity. To address this challenge, we propose a mutation-based method integrated with Bayesian optimization to detect bugs in FPGA logic synthesis compilers. The method, called SmootHDL, consists with three key components, the Test-program Generation Component (TPG), the Bayesian Diversity Selection Component (BDS), and the Equivalent Test Check Component (ETC). SmootHDL starts with a set of test cases which are called original test cases. To expand the timing test space, the TPG use a control-flow mutation strategy to generate new test cases which is functional equivalent with the original test cases. Furthermore, for increasing diversity of the generated test cases, the BDS combine the Bayesian optimizer to construct a selector based on the control-flow diversity. Finally, the ETC identifies bugs by comparing simulation outputs between original and generated test cases. Since the mutations preserve functional equivalence, functional equivalent test case should have same output. When SmootHDL detects any discrepancy between the outputs, it indicates a potential compiler bug. SmootHDL then reproduces the inconsistent behavior and reports the bug to official developers. We evaluated our method on the mainstream FPGA logic synthesis compilers. Experiments show that SmootHDL can generate more diverse test cases and detect more bugs compared with the latest testing methods for FPGA logic synthesis compilers. Over three months, SmootHDL found 16 bugs, 12 of which were confirmed by official technical support, and 6 have already been fixed.