Bootstrapped Control Limits for Score-Based Concept Drift Control Charts

Monitoring for changes in a predictive relationship represented by a fitted supervised learning model (i.e., concept drift detection) is a widespread problem in modern data-driven applications. A general and powerful Fisher score–based concept drift approach was recently proposed by Zhang et al. (2023), in which detecting concept drift reduces to detecting changes in the mean of the model’s score vector using a multivariate exponentially weighted moving average (MEWMA). To implement the approach, the initial data must be split into two subsets. The first subset serves as the training sample to which the model is fit, and the second subset serves as an out-of-sample test set from which the MEWMA control limit (CL) is determined. In this paper, we retain the same score-based MEWMA monitoring statistic as Zhang et al. (2023) and develop a novel nested bootstrap procedure computing a much more accurate CL that also allows the entire initial data to be used for model fitting, thereby yielding a more accurate baseline model while eliminating the need for a large holdout set. The outer bootstrap loop is fully parallelizable, making the method computationally practical, with CL setup times comparable to those of Zhang et al. (2023). We show that a standard nested bootstrap substantially underestimates the variability of the monitoring statistic and develop a 0.632-like correction that appropriately accounts for this. We demonstrate the advantages with numerical examples.