PhysCISC: Code and dataset for process-credibility weighted self-consistency voting in physics reasoning

# PhysCISC Dataset and Reproducible Code This release accompanies the accepted paper: > **PhysCISC: A Process-Credibility Weighted Self-Consistency Voting Framework for Physics Reasoning with Large Language Models** It contains the released datasets and the code needed to rebuild the dataset-processing workflow, run the three-model/four-dataset experiments, and regenerate the analysis tables. Generated model outputs are intentionally not bundled; the commands below create them under `outputs/`. ## Contents - `data/physcisc_main_zh_intersection_3848.json`: 3,848-question Chinese PhysCISC benchmark. - `data/physcisc_main_zh_intersection_3848.jsonl`: equivalent JSONL representation. - `data/physcisc_main_zh_intersection_3848_indices.json`: traceability indices and counts. - `data/external_benchmarks/`: the exact standardized MMLU-Physics, SciBench-Physics MCQ, and UGPhysics MCQ files used by the runner. - `code/dataset_pipeline/`: scan, parse, rewrite, filter, label, intersection, and public-benchmark conversion code. - `code/experiments/`: chain sampling, answer parsing, P(True), 0--4/0--9 process-credibility scoring, and JSONL logging. - `code/analysis/`: accuracy, paired tests, WQD/AUROC, sample efficiency, temperature, difficulty, domain/type, complementarity, cost, and human-audit sampling code. - `docs/REPRODUCIBILITY.md`: exact end-to-end commands and expected output layout. ## Tested environment - Python 3.10 - PyTorch 2.8.0 - Transformers 4.57.6 - Windows 11 Pro for Workstations - Intel Core i9-13900K, two NVIDIA GeForce RTX 4090 GPUs, 128 GB RAM Model identifiers: - `Qwen/Qwen2.5-7B-Instruct` - `meta-llama/Llama-3.2-3B-Instruct` - `mistralai/Mistral-7B-Instruct-v0.3` Local model directories may be supplied through environment variables; see `docs/REPRODUCIBILITY.md`. ## Quick verification From the package root on Windows PowerShell: ```powershell py -3.10 -m venv .venv .\.venv\Scripts\python.exe -m pip install -r requirements.txt .\.venv\Scripts\python.exe .\code\experiments\run_experiments.py --dry-run .\.venv\Scripts\python.exe .\code\analysis\reproduce_analysis.py --dry-run ``` The experiment dry run verifies the four dataset paths without loading model weights. The analysis dry run expects the 12 JSONL files produced by a complete experiment run and reports each path as present or missing. ## Full reproduction Run each model, retaining the default run ID `physcisc_v2`: ```powershell $env:CUDA_VISIBLE_DEVICES="0" .\.venv\Scripts\python.exe .\code\experiments\run_experiments.py --models qwen2_5_7b_instruct .\.venv\Scripts\python.exe .\code\experiments\run_experiments.py --models llama3_2_3b_instruct .\.venv\Scripts\python.exe .\code\experiments\run_experiments.py --models mistral_7b_instruct_v0_3 ``` Then regenerate the analyses: ```powershell .\.venv\Scripts\python.exe .\code\analysis\reproduce_analysis.py ``` Experiment JSONL files are written to `outputs/experiments/physcisc_v2/ / .jsonl`; analyses are written to `outputs/analysis/`. Generation is stochastic (`temperature=0.7`, `top_p=0.95`), so a fresh run reproduces the released evaluation protocol but is not expected to be bit-for-bit identical to a previous sampled chain pool. All comparisons within a run use prefixes of the same stored 32-chain pool. ## Release boundary Raw book PDFs, scanned pages, page images, private API credentials, model weights, caches, and generated experiment outputs are not included. Re-running the full book-to-dataset construction pipeline requires legally obtained source scans and the API credentials described in `code/dataset_pipeline/README.md`. The final standardized datasets included here are sufficient to rerun the reported evaluation protocol. ## Citation and licenses See `CITATION.cff`, `LICENSE`, `LICENSE-CODE`, `LICENSE-DATA`, and `docs/DATA_PROVENANCE_AND_LICENSES.md`.