Does the Tool Matter? Exploring Some Causes of Threats to Validity in Mining Software Repositories

The proposed contribution is based on a paper published in the proceedings of the 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) [Ho25]. The reproduction package is publicly available at Zenodo: (10.5281/zenodo.14091454). Mining software repositories is a common means for empirical software engineering, making suitable tools an indispensable prerequisite. Ideally, valid results should not depend on technical details of data collection and processing. However, the complexity of pipelines raises the question if (and to what extent) tools agree on their results. The study employs two tools for data extraction and developer network construction and quantitatively and qualitatively compares outcomes to address this concern. The results demonstrate that despite similar trends, even simple metrics such as the number of developers may differ substantially. We show technical changes to overcome these issues and summarise lessons learned to help others avoid common pitfalls.