health-portfolio: Severity-Weighted OR-Gate Evaluation of Clinical AI Ensembles

Background. Clinical AI screening models are typically evaluated by the area under the receiver operating characteristic curve (AUC), which treats all classification errors as equally costly. When health systems deploy multiple AI models, no established framework exists for determining whether the ensemble provides genuine safety value or merely replicates the primary model’s error profile. Methods. We trained 11 deep learning models (DenseNet121, EfficientNet-B3, ResNet50, ViT-Base, and RETFound, each in binary and 5-class severity formulations, plus one adversarially decorrelated variant) on 13,532 EyePACS diabetic retinopathy images and evaluated all 55 pairwise ensembles on two independent test sets: EyePACS (n = 2,900; 130 severe cases) and APTOS (n = 3,662; 488 severe cases). We compared three evaluation approaches: AUC-based ensemble averaging, OR-gate deployment (flag if either model predicts referable), and a severity-weighted cost framework derived from ETDRS progression rates and quality-adjusted life year (QALY) data. A threshold-sweep analysis compared OR-gate ensembles against optimally thresholded single models at matched operating points. A 2×2 factorial decomposition separated the contributions of the aggregation rule (averaging vs. OR-gate) and evaluation metric (AUC vs. severity-weighted cost). Weight robustness was assessed across five clinically plausible severity-cost scenarios. Bootstrap resampling (n = 1,000) provided 95% confidence intervals for all key quantities. Results. Under AUC-based averaging, 51 of 55 pairs showed zero or negative ensemble benefit. Under OR-gate severity-weighted evaluation, the best ensemble (DenseNet121 5-class + EfficientNet 5-class) reduced severity-weighted cost by 39% on EyePACS (solo cost 6,468 [95% CI: 4,435–8,812] vs. OR-gate cost 3,972 [2,820–5,554]; reduction 2,495 [976–4,360]) and 68% on APTOS. The 2×2 decomposition showed 96% of ensemble benefit derived from the OR-gate aggregation rule (rule effect = −2,410) versus 4% from the severity metric (metric effect = −112). Threshold-sweep analysis confirmed that OR-gate ensembles dominate optimally thresholded single models: at matched severe-miss rates, the OR-gate required 173 false positives versus 603 for the single model. Error correlation (ρ) predicted concordant-miss risk (r = 0.665 [0.456–0.797]), with exploratory evidence for a threshold near ρ ≈ 0.70. The top-ranked ensemble was robust across all five clinically plausible weight scenarios, changing rank only under equal weighting of all error types (a clinically implausible assumption). Architecture diversity was the primary driver of low ρ; adversarial training and task-formulation diversity provided smaller, inconsistent benefits. Conclusions. The choice of evaluation metric and aggregation rule jointly determine whether clinical AI ensembles appear worthless or transformative. Severity-weighted OR-gate evaluation detects clinically important complementarity that AUC-based methods miss. We propose error-correlation disclosure and concordant severe-miss analysis as standard components of multi-AI screening program evaluation.