CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents

Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by the code they can reliably generate from parametric knowledge alone.As a result, current systems either mutate a small number of manually-crafted experiment examples, or operate solely from parametric knowledge, limiting quality and reach.We introduce CODEDISTILLER, a system that automatically distills large collections of scientific GITHUB repositories into a vetted library of working domain-specific code examples, allowing ASD agents to expand their capabilities without manual effort.Using a combination of automatic and domain-expert evaluation on 250 materials science repositories, we find the best model is capable of producing functional examples for 74% of repositories, while our downstream evaluation shows an ASD agent augmented with a CODEDISTILLER generated library produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples.We also evaluate LLM-AS-A-JUDGE ratings against domain-expert ratings in an A/B testing paradigm, finding moderate agreement and suggesting that inexpensive proxy metrics may be feasible for evaluating scientific discovery systems at scale. 1