Artifact for "Assumption Mining: Extracting and Updating Implicit Requirements in LLM Code Generation"

Read full text Summarise with VIRAT
Authors: Anonymous
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Replication Package for the Paper: AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation, Jie JW Wu. Submitted to IEEE Transactions on Software Engineering. When an LLM generates code from a natural-language prompt, it silently commits to many implicit decisions—including input validation, error handling, return types, persistence, algorithmic choices, and security posture—that are never explicitly stated in the prompt. AssumptionMiner: Extracts these implicit assumptions from a (prompt, code) pair as structured records using a six-category taxonomy (T1–T6). Maps each assumption to the code regions that implement it through a tree-sitter AST analysis combined with keyword narrowing. Incrementally regenerates only the affected code regions when a developer revises an assumption, leaving the remainder of the code unchanged. This replication package contains everything needed to: Inspect the benchmark and the raw results underlying every number reported in the paper. Re-run all offline analyses without requiring an API key or GPU. Re-run the full LLM experiments using either an OpenAI API key or a local vLLM server. Contents assumption_miner/ — Core library containing one module for each paper component (C1–C5), the CE/KBE/DG/CQ/CoT baselines, the AssumptionRecord and CodeRef schemas, and a 119-test offline test suite. data/ — The AssumptionMiner benchmark (180 LLM-generated tasks containing 676 assumptions), including a curated subset of 30 tasks with 109 human-verified assumptions. Also includes data/results/, which contains the raw JSON result files used to generate every table in the paper. scripts/ — Experiment and analysis scripts. Running gen_paper_tables.py regenerates every RQ1, RQ2, and RQ3 table, including confidence intervals, directly from the raw results. verification/ — Complete materials from the two-rater human verification study, including the annotation protocol, both raters' independent annotation sheets, adjudication decisions, the reconciled consensus.jsonl gold reference, and the 30 labeled prompt–code pairs used to calibrate the strict decision-level metric (τ = 0.56). interface/ — Web-based interactive interface. README.md — Repository overview, contents map, results-to-paper traceability table, benchmark provenance, and step-by-step reproduction instructions. Reproducibility All tables in the paper can be regenerated offline from the included raw results by running: python scripts/gen_paper_tables.py Local-model experiments were conducted using an NVIDIA RTX PRO 6000 Blackwell Max-Q GPU (96 GB VRAM). Model temperatures are fixed in the implementation: 0.0 for assumption extraction and 0.2 for code generation and regeneration. This package is released for research use.

Fields of Study & Subjects

Software Engineering Research Software Engineering Techniques and Practices Software Testing and Debugging Techniques

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
27 March 2026
Language
EN