Automated Generation of Issue-Reproducing Tests by Combining LLMs and Search-Based Testing

The full version of this paper is accepted to the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025). A software issue is typically an online report that describes a bug or a feature request, occasionally including details like stack traces or expected/observed behavior [Be08]. In most cases, developers address these issues through a set of code changes (also known as a patch). In this context, an issue-reproducing test is a test accompanying the patch that fails on the unpatched code (validating the presence of the issue) and passes on the patched code (validating its resolution). An issue-reproducing test increases confidence that the issue (1) can be replicated and (2) will not be reintroduced in the future. Writing issue-reproducing tests is a demanding and time-consuming task [LIH17] that is often overlooked by developers [LY17], thus recent work has investigated ways to automatically generate these tests to support developers. In particular, recent techniques have proposed the use of Large Language Models (LLMs), either in a zero-shot setting [Ah24] or in a multi-step workflow where the output of the previous LLM call is used as input to the next one [Ah24; Na25]. However, relying solely on LLMs for the generation of tests is prone to hallucinations [EP24], leading for example to tests trying to import non-existent modules or use non-existent methods. In addition, recent work does not yet leverage information that could be useful for the generation of the issue-reproducing test such as the focal context or the structure of existing tests for the same class, which could hint the test setup or the available mocks.