Abstract
Early Warning and Regime Detection for Ammonia Events at the Shafdan Wastewater Treatment Plant: A Machine Learning Study on Industrial This dataset and accompanying analysis document a machine learning research project conducted on operational sensor data from the Shafdan wastewater treatment plant (WWTP) in Israel — one of the largest reclaimed water facilities in the Middle East, serving the greater Tel Aviv metropolitan area. The plant operates a large-scale activated sludge process for secondary treatment, producing reclaimed water used extensively in agricultural irrigation. The central research objective was to determine whether ammonia breakthrough events — exceedances of approximately 15.2 mg/L in the effluent of the biological treatment stage — can be predicted from process sensor data with a practically useful lead time of up to six hours. A secondary objective was to characterise the macro-level operational states of the plant and to develop a two-level warning architecture in which a macro-state detection layer gates a short-term event prediction layer. The study draws on approximately 2.5 years of continuous 10-minute resolution process data (November 2017 – June 2020), comprising 68 raw CSV files, 881,828 measurement rows, and 86 process variables. Key sensor streams include dissolved oxygen (DO) in multiple reactor zones, influent and recirculation flow rates, pH, turbidity, mixed liquor temperature, rotor depth levels, secondary clarifier sludge levels, waste activated sludge (WAS) flow, and energy counters. The target variable is the ammonia concentration measured at the effluent of the biological treatment stage (Stage B). The research follows an iterative, diagnostic approach across eight analytical phases. Early phases focus on data cleaning, feature engineering, and the identification of multivariate precursor patterns preceding ammonia events via lead-lag audit. A critical finding from this phase was that the plant does not behave as a homogeneous stationary process: ammonia events are clustered within extended macro-active operational periods, and a universal classifier applied globally fails to generalise across the train-test boundary due to pH-driven regime separation between the 2017–2019 training era and the 2020 test period. Subsequent phases develop and evaluate a two-level architecture. The upper layer (Stage 1) is formulated as a regime-shift transition-risk score based on distance from a stable-inactive process baseline, evaluated across multiple smoothing and threshold configurations. The lower layer (Stage 2) is a binary classifier trained exclusively within confirmed macro-active periods, predicting whether an ammonia event will occur within the next six hours. The Stage 2 model achieves ROC-AUC of 0.843 and PR-AUC in the range 0.853–0.898 on the chronological holdout, with 84% event coverage and a 3.36× system lift within active periods. The Stage 1 gate, when tested end-to-end, did not achieve operationally sufficient coverage in the 2020 test period (2.6% event coverage under the primary configuration). Diagnostic notebooks attribute this primarily to an operational regime shift in the plant's 2020 behaviour — characterised by lower dissolved oxygen volatility and a systematic pH decline — rather than to total feature-space separation from the training distribution. A minimal recalibration of the Stage 1 score to a 2020-specific stable-inactive baseline improved event coverage approximately fourfold (from 2.3% to 9.8%), suggesting annual recalibration as a necessary operational requirement. A supplementary audit examines whether a portion of the Stage 2 false positives may reflect operator-prevented events in a closed-loop control environment. A proxy-based Prevention Match Rate (PMR) of 30.8% was observed, with a statistically significant excess above the background intervention rate. However, cross-correlation analysis showed that proxy intervention signals co-occur with model score peaks rather than following them, precluding a confirmed reactive-operator interpretation. The absence of direct operator intervention logs represents the primary unresolved limitation of the study. The work contributes a reproducible analytical pipeline for hierarchical ammonia warning on industrial WWTP time series, a documented regime-shift scoring methodology, and a frank assessment of the conditions — particularly operator log integration and annual score recalibration — under which a two-level automated warning system can be considered operationally viable. Keywords: wastewater treatmentWWTPammonia predictioneffluent quality predictionearly warning systembiological nitrogen removalactivated sludge processdissolved oxygenprocess monitoringanomaly detectionregime shift detectiontwo-level architecturehierarchical classificationrare event predictionimbalanced time seriesindustrial machine learningpredictive maintenancesensor datamultivariate time seriesfeature engineeringwalk-forward validationconcept driftoperational regime changerecalibrationprevention match rateclosed-loop controloperator interventionwater reusereclaimed waterShafdan