Dilemma between bias interaction and trustworthy AI in human–AI collaborated judgments

In today’s world, humans collaborate with artificial intelligence (AI) and often make judgments with the help of AI assistance in such contexts as medical diagnoses. Greater AI accuracy and trustworthiness is generally regarded as the goal of designing decision-support AI systems. However, highly accurate AI may not always provide optimal assistance. Rather, AI with bias, especially in the direction opposite to an individual’s bias, may improve the accuracy of individual judgment. This is because the AI’s bias will cancel out the individual’s bias (e.g., individuals’ overestimation bias is corrected by AI’s underestimation bias). In this study, we call such aggregation of human and AI biases resulting in a change of final judgment “bias interaction.” We investigated such interactions using a simple perceptual judgment task simulating medical judgments. First, a theoretical analysis using computer simulations showed that optimal AI assistance depended on individuals’ biases. Second, two behavioral experiments demonstrated that AI with biases in the direction opposite to participants’ biases tended to improve participants’ accuracy. However, participants tended to evaluate such AI as being less trustworthy. The theoretical and empirical results of our study raise questions about conventional beliefs that attribute superiority to more accurate and trustworthy AI. We also discuss the practical implications of designing AI to be a better collaborator.