AutoOracle: High-Quality C++ Test Oracle Generation via Data Quality-Driven and Filtering-Enabled LLMs

Test oracles (assertions used to verify program behavior) serve as a validation benchmark in software testing by comparing the actual outputs with expected results. However, manually building test oracles is time-consuming, and the existing AI-driven test oracle generation schemes often suffer from noisy training data and unreliable inference outputs. To this end, this paper proposes AutoOracle, a novel framework that leverages large language models (LLMs) for automated test oracle generation in C++. It automatically constructs the first large-scale, high-quality C++ training dataset through Code De-Differentiation Method and Data Quality Scoring and Filtering Mechanism, fine-tunes the first domain-specific LLM for C++ test oracle generation, and improves the reliability of generated oracles via a Suspicious Oracle Filtering Framework. We have evaluated our approach in terms of oracle quality and practicality. Experimental results on open-source C++ projects demonstrate that AutoOracle improves accuracy by 2.13 times and reduces the error rate by 52.3% compared with the state-of-the-art (SOTA) method. In practice, on an embedded C++ solid state drive firmware project at Samsung Electronics, AutoOracle generated test oracles with higher compile/runtime success rates and received better feedback in a survey of 19 practitioners, compared with the SOTA method. These results indicate that AutoOracle can generate high-quality test oracles, thereby significantly boosting test efficiency.