Deference-Aware Evaluation for Human-in-the-Loop AI Systems

Read full text Summarise with VIRAT
Authors: Yuyu Shen
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Standard AI evaluation frameworks penalise calibrated uncertainty — creating systematic pressure toward overconfident AI in exactly the contexts where human oversight matters most. This paper proposes Deference-Aware (DA) Evaluation: the first unified framework for HITL AI classifiers that credits appropriate deferrals as correct rather than penalising them. Instantiated in systematic review evidence synthesis across four evaluation tiers and nine frontier models. Yuyu Shen | Hopperlace Research | March 2026 Related product: https://www.evidencesynthesisai.com

Fields of Study & Subjects

Ethics and Social Impacts of AI Human-Automation Interaction and Safety Explainable Artificial Intelligence (XAI)

Publication details

Publication date
27 March 2026
Language
EN
Document type
Preprint