Using a Sledgehammer to Crack a Nut? Revisiting Automated Compiler Fault Isolation

Background: Compilers are fundamental to software development, translating high-level source code into executable software systems. Faults in compilers can have severe consequences and thus effective localization and resolution of compiler bugs are crucial. Problem: In practice, developers often examine version history to identify and investigate bug-inducing commit (BIC) for fixing bugs. However, while numerous sophisticated Spectrum-Based Fault Localization (SBFL) techniques have been proposed for compiler fault isolation, their effectiveness has not been evaluated against the BIC-based strategies widely adopted in practice. Objective: This study aims to bridge this gap by directly comparing a BIC-based strategy, called Basic, with representative SBFL-based compiler fault isolation techniques. The BIC-based strategy closely aligns with common developer practices, as it directly identifies the BIC and treats the files modified in that commit as faulty candidates. Method: Basic strategy identifies the most recent good release and earliest bad release, and then employs a binary search to pinpoint the bug-inducing commit. All files modified in the identified commit are flagged as potentially faulty. We rigorously compare Basic against SBFL techniques using a benchmark consisting of 60 GCC bugs and 60 LLVM bugs. Result: Our analysis reveals that Basic generally outperforms state-of-the-art SBFL techniques on the critical Top-1 and Top-5 ranking metrics. Conclusion: This study provides new insights into the practical effectiveness of SBFL techniques in real-world compiler debugging scenarios. We recommend that future research adopt Basic as a baseline when developing and evaluating new compiler fault isolation methods.