Less Is More: Failing Test Generation with Large Language Models

Failing test generation is challenging. It involves searching in a vast space for fault-triggering test inputs and the oracles asserting these faulty executions. Despite techniques proposed to generate tests using Large Language Models (LLMs) , they are ineffective in finding failing tests, particularly for programs that implement non-trivial coding tasks such as medium/advanced-level coding contest problems. To tackle this limitation, we are inspired by an earlier finding that constituent snippets within a program typically implement simpler coding tasks compared to the program as a whole. As a result, LLMs can be leveraged to generate failing tests that target a program’s constituent snippets, thereby revealing the program defects. Leveraging this insight, we propose Microscopic Test Generation (M it G en ) , a novel technique of failing test generation. Unlike previous approaches that generate tests to fulfill code coverage, MitGen focuses on generating tests that reveal faults in a given program’s constituent code snippets. We evaluate MitGen using Starcoder2-15B-instruct-v0.1 , Meta-Llama-3-8B-Instruct , and CodeQwen1.5-7B-Chat on two popular benchmarks ( EvoEval-Difficult and ClassEval ) and 100 real-world subjects. We compare MitGen with three baselines, including state-of-the-art approaches ( Differential Prompting and Pynguin ), in finding failing tests. The evaluation results show that MitGen ’s recall is 0.66, a 112.7% enhancement over the best baseline (0.31).