From Bugs to Benchmarks: A Comprehensive Survey of Software Defect Datasets
Abstract
Software defect datasets, which are collections of software bugs, are essential resources to facilitate empirical research and enable standardized benchmarking for a wide range of software engineering techniques, including emerging areas like agentic AI-based software development. Over the years, numerous software defect datasets have been developed, providing rich resources for the community, yet making it increasingly difficult to navigate the landscape. This article provides a comprehensive survey of 151 software defect datasets, covering their scope, construction, availability, usability, and practical uses. We also suggest potential opportunities for future research based on our findings, such as addressing underrepresented kinds of defects. A complete catalog of all surveyed software defect datasets is available at https://defect-datasets.github.io/ .
Fields of Study & Subjects
Software Engineering Research
Software Testing and Debugging Techniques
Software Reliability and Analysis Research
Publication details
- Journal / source
- ACM Computing Surveys
- Publisher
- Association for Computing Machinery (ACM)
- Volume
- 58
- Issue
- 10
- Pages
- 1–39
- Publication date
- 11 February 2026
- Language
- EN