Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection

A software engineering issue (SWE issue) is easier to resolve when accompanied by a reproduction test. Unfortunately, most issues do not come with functioning reproduction tests, so this paper explores how to generate them automatically. One technique that helps with this task is inference scaling, but traditional temperature-based scaling tends to lack diversity. This paper introduces heterogeneous prompting to address this problem. Another technique that helps with this task is execution feedback, but this is hampered by the fact that the new code (after issue resolution) to execute does not yet exist. This paper introduces novel approaches to get around that problem. We implemented our techniques in a new reproduction test generator called e-Otter++. Experiments show that e-Otter++ represents a leap ahead in the state-of-the-art for this problem, generating tests with an average fail-to-pass rate of 63% on the TDD-Bench Verified benchmark.