Multilevel, Multireferent Trust, Communication, and Performance in Human–Agent Teams: Evaluating the Effects of Agent Reliability Pairing

Abstract Automation is pervasive, and humans often collaborate with it to perform various tasks. With technological advancements, it has become increasingly important to understand the characteristics of complex teams composed of multiple humans and multiple agents. In this study, we examined various team-level metrics in multioperator, multiagent teams, including trust, communication, and task completion time (TCT). We conducted a human subjects experiment with 30 teams, each consisting of 2 human and 2 agent players, performing collaborative block-moving tasks. Each team completed 10 trials under 3 agent reliability pairing conditions: perfect (both agents had 100% reliability), mixed (1 agent had 100%, the other 50%), and imperfect (both agents had 50% reliability). Humans searched for target-colored blocks and issued commands to the agents, which followed predetermined logic to move the blocks to the drop zone. Participants’ trust ratings for each teammate and for the team were collected after each trial. From the experimental data, we quantified several team metrics: team-level trust in each referent (each teammate and the team), divergence of trust in each referent, team communication frequency, commands per block delivery, divergence of team communication, and TCT. We evaluated the effect of agent reliability pairing on each of these metrics and explored the associations between them. The results indicate strong interdependence: Shorter TCT is associated with higher team-level trust in team, greater agreement in trust ratings, and more effective, convergent communication patterns shared by the humans. This study offers important guidelines for evaluating human–agent teams at a higher, team-level perspective.