Glitches in the matrix: threats to the validity of data-driven business process simulation

Abstract Business Process Simulation (BPS) is commonly used to conduct what-if analyses and assess the impact of changes to processes and their underlying information systems. In a typical data-driven BPS approach, a simulation model is derived from an event log and evaluated through a six-step method. However, following these steps can introduce systematic errors that potentially bias evaluation results and, thereby, lead to incorrect assessments of simulation model quality. To address this, we identify five systematic errors as threats to the internal validity of evaluation results, highlighting the problem and forming the central contribution of this paper. Through empirical examples, we demonstrate that these threats can distort evaluation outcomes, both individually and in combination, even when the same process and simulation model are applied. Building upon this analysis, we outline a conceptual framework that integrates these insights with established methods from related fields to guide mitigation of the identified threats. By recognizing the threats, we highlight how simulation accuracy may be misjudged, and by proposing this framework, we enable more valid evaluations that better support decisions about redesigning business processes and information systems.