Reducing Alert Fatigue Through AI Ranking: A Deployed Public Health Data Monitoring System

Abstract Public health experts need scalable methods to monitor large volumes of health data and identify data points that may indicate significant events, such as outbreaks or data quality issues. Experts then triage and analyze these data points so that they can prevent downstream errors in forecasting or policy. Still, traditional alert‐based data monitoring systems, used for decades in public health practice, may return tens of thousands of unhelpful alerts, making relevant events difficult to identify. We introduce a human‐in‐the‐loop AI system for public health data monitoring that uses a ranking‐based AI anomaly detection method. This system has since been deployed at a national public health organization under loads of up to 5 million data points daily. A three‐month longitudinal deployment evaluation revealed a significant improvement in system goals, including a 54x increase in data reviewer efficiency and increased engagement compared to traditional alert‐based methods.