Exploring the Requirements of Clinicians for Transparent and Trustworthy Decision Support Systems in Intensive Care: Semistructured Interview Study (Preprint)

BACKGROUND Understanding how digital systems can support clinical decision-making is crucial, especially with the growing deployment of increasingly complex artificial intelligence (AI) models. This complexity raises concerns about trustworthiness, impacting the safe and effective adoption of such technologies. In intensive care units (ICUs), where clinicians make high-stakes, time-sensitive decisions, decision-support tools must be designed to align with clinical needs and cognitive workflows. Improved understanding of decision-making processes and requirements for decision support tools is vital for providing effective solutions. OBJECTIVE This study aimed to investigate ICU clinicians’ decision-making processes, the challenges posed by patient complexity, and the requirements for decision-support systems to ensure transparent and trustworthy recommendations. METHODS We conducted group interviews with seven ICU clinicians, representing diverse roles and experience levels, to explore perspectives on decision-support tools. Reflexive thematic analysis was used to identify key themes and thereafter design recommendations. RESULTS Three core themes emerged from the analysis: (T1) ICU decision-making relies on a wide range of factors; (T2) patient complexity challenges shared decision-making, and (T3) acceptability and usability of decision support systems. Design recommendations derived from clinical input provide insights to inform future decision support systems for intensive care. CONCLUSIONS Decision-support tools have the potential to enhance ICU decision-making, but their adoption depends on alignment with clinicians' needs and workflows. To improve trust and usability, future systems must be transparent in their recommendations, adapt to varying patient complexities, and facilitate, rather than replace, human expertise. Our findings inform the development of digital systems that are both transparent and trustworthy, aiding clinically acceptance in ICU settings. CLINICALTRIAL Not applicable.