Beyond Binary Verdicts: Aleatoric Uncertainty Quantification in Agentic CI/CD Quality Pipelines

Multi-agent CI/CD quality pipelines that render binary pass/fail verdicts provide no signal about verdict certainty. A passing verdict from a low-confidence agent is indistinguishable from one issued with high confidence — a gap this paper terms the false-safe silence problem. This paper introduces per-agent confidence propagation to the proposed agentic AI QA system, equipping eight specialist agents with a self-assessed confidence score alongside each verdict. Scores are aggregated with DeepEval and RAGAS metrics into a unified uncertainty score activating a tiered response: a verdict downgrade to PASS_UNCERTAIN for medium-uncertainty cases and a second HITL escalation path for high-uncertainty cases. Two extensions deepen the mechanism: a correlation-aware aggregation layer (Feature F6) that reduces false-positive escalation rate from 6.2% to 4.8%, and an isotonic regression recalibration layer (Feature F5) that reduces Expected Calibration Error from 0.14 to 0.08. Simulation over 500 synthetic pipeline runs calibrated to a 120-PR empirical dataset demonstrates that the confidence gate surfaces 18.4% of passing verdicts as high-uncertainty and 15.8% as medium-uncertainty (PASS_UNCERTAIN), extending uncertainty coverage from 18.4% to 34.2% while preserving CI throughput.