TOGBench artifact: AIware-2026

TOGBenchTOGBench is a benchmark suite for automated test oracle generation (TOG) built around OE25dev, a multi-variant dataset mined from developer-written unit tests across 25 real-world Java projects spanning 56 modules.Unlike prior benchmarks that rely on automatically generated tests or restrict evaluation to single assertions, TOGBench preserves realistic test context, six oracle categories, and supports end-to-end executable evaluation through oracle injection, compilation, execution-driven validation, and mutation-based adequacy assessment.OverviewProjects: 25Modules: 56Oracle types: 6Language: Java 8FeaturesOE25dev_single: 77,299 single-oracle instances derived from developer-written tests. Each instance contains one masked oracle block with the surrounding test prefix preserved.ASSERTION_ONLY: 73,023MUST_THROW: 2,899MUST_NOT_THROW: 694IF_THROWN_ASSERT: 257MUST_THROW_WITH_PROPERTIES: 319FAIL_ONLY: 107OE25dev_multiple: 15,272 original test methods containing at least two standard oracle blocks of the same family, released without decomposition.OE25dev_mixed: 1,576 original test methods containing at least one assertion oracle and at least one exception oracle, released without decomposition.OE25dev_custom: 5,747 instances of project-defined assertion wrappers and helper methods recorded as individual oracle instances.Quick StartRequirements: Python 3.9+, Java 8 JDK, Maven 3.xStrip ground-truth oracles: python scripts/remove_assertion.pyInject your TOG tool predictions: python scripts/inject_assertion.pyRebuild test files: python scripts/rebuild_dataset_tests.pyRun Maven and collect results: ./scripts/clean_run_mvn.shOr run the full pipeline at once: ./scripts/run_project_dataset.shOutputsAll results are written to evaluation/evaluation_results/.Compilation Rate — proportion of reintegrated test classes that compile successfullyExecution Pass Rate — proportion of compiled tests that passFalse Positive Rate — proportion of passing tests that pass on incorrect program versionsMutation Score — oracle adequacy measured via PIT mutation testing on green suitesScriptsrun_project_dataset.sh — full pipeline entry pointrebuild_dataset_tests.py — reconstructs .java test files into projects_decomposed/inject_assertion.py — merges predictions into input CSV and produces inputs_llm.csvremove_assertion.py — strips ground-truth oracles and produces inputs_no_assert.csvclean_run_mvn.sh — runs Maven across all modules and saves logs to logs/comment-incompatible_assertions.py — comments out assertions causing compilation errorsdata_csv/test_count.py — aggregates per-project test pass/fail counts