Designing for Oversight: An Empirical Investigation of the Dual Impact of AI Dependency and Information Abstraction on Human Supervision in Decision-Making Teams

The proliferation of AI capabilities and growing appetite in workflow integration warrants a closer look at how humans supervise multiple AI teammates. Through a mixed-methods (N = 80) study, we explored how dependency between agents and abstraction of task-related information influence humans’ ability to monitor and utilize AI assistance effectively. Results revealed an asymmetry in user perceptions of AI arising out of their dependency conditions. These perceptions were not driven by how well users understood their AI teammates, but by their (limited) sensitivity to AI’s error severity. This led to masking of less severe errors, especially when the agents depended on one another. Dependencies also led to heightened workload, which was not mitigated by the level of information abstraction. However, users using less abstracted data appreciated AI assistance better, but were low on confidence and performance satisfaction. Through these findings, we offer renewed perspectives on designing multi-agent sociotechnical systems for collaborative decision-making.