U-TraCE: a conformal prediction approach to uncertainty quantification in black-box models

Abstract Modern machine learning (ML) systems, particularly deep neural networks, have become increasingly complex and opaque, making reliable uncertainty quantification (UQ) essential. Conventional UQ methods often prove inadequate for black-box model evaluation scenarios, either because model internals are not accessible or because the resulting uncertainty is expressed in terms of a related predictor rather than the model under evaluation. We introduce U-TraCE (Uncertainty Tracking for Complex Estimators), a novel UQ method built on the Conformal Prediction (CP) framework that provides formal and traceable uncertainty bounds for opaque ML systems. Unlike existing CP approaches, which describe uncertainty through confidence intervals or prediction set sizes at a fixed uncertainty level, U-TraCE yields an explicit numerical bound on the error probability of the standalone black-box model. U-TraCE was evaluated on diverse tasks, including MNIST classification and cardiac magnetic resonance imaging (MRI) segmentation, representing both benchmark and real-world domains. Across these settings, it produced conservative yet well-calibrated uncertainty bounds that closely follow empirical uncertainty, even with limited calibration data. By bridging theoretical UQ and operational model validation, U-TraCE establishes a foundation for objective and automated conformity assessment of complex ML systems, supporting trustworthy deployment in safety-critical environments.