ase262205
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
# Artifact for ASE 2026 Submission# 2205 This repository contains the analysis scripts and configuration files used in our study. ## Overview The artifact supports the empirical analyses reported in our experience paper. It covers the following: - single-LLM correctness validation- repeated querying across three runs- aggregation by majority voting and strict agreement- 2-LLM and 3-LLM ensemble analyses- generation of plots for the reported results The study used the publicly available ScienceQA dataset from the official ScienceQA repository:https://github.com/lupantech/ScienceQA ## Repository Structure ```textscripts/ analysis/ aggregate.py dedup_utils.py experiments.py io_utils.py main.py metrics.py parse_traits.py plots.py taxonomy_utils.py config/ model_taxonomy.csv``` ## Main Components - ```main.py``` manages and initiates the analysis pipeline and plots generation.- ```io_utils.py``` loads the positive and negative result splits.- ```parse_traits.py``` extracts binary correctness decisions.- ```experiments.py``` runs the single-LLM, repeated-querying, and ensemble analyses.- ```aggregate.py``` implements aggregation rules such as majority voting and strict agreement.- ```metrics.py``` computes the evaluation metric.- ```plots.py``` generates the plots reported in the paper.- ```taxonomy_utils.py``` loads the LLM taxonomy and supports grouping by provider, family, and tier.- ```model_taxonomy.csv``` defines the LLM metadata used in the analyses. ## Expected Inputs The analysis expects a results/ directory at the repository root with the following structure: ```textresults/ positives/ negatives/``` These directories should contain the prediction outputs for the evaluated LLM judges across the three runs used in the study. The configuration file is expected at: ```textscripts/config/model_taxonomy.csv``` ## What the Pipeline Does The pipeline performs the following steps: 1. loads positive and negative prediction outputs2. filters them to the prompt configuration used in the study3. extracts binary correctness labels4. computes single-run and repeated-querying results5. computes 2-LLM and 3-LLM ensemble results6. generates the plots reported in the paper ## Running the Analysis From the repository root, run: ```textpython scripts/analysis/main.py``` ## Generated Outputs The analysis writes its outputs under:```textresults/analysis/```This includes: * ```results_overview.csv```* ```ensemble_results.csv```* generated plots under ```results/analysis/figures/``` ## Anonymity Notice```textThis repository has been prepared for anonymous review.```
Fields of Study & Subjects
Topic Modeling
Natural Language Processing Techniques
Scientific Computing and Data Management
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 30 March 2026
- Language
- EN