Ethical Assessment of Generative AI Tools for Clinical Summarization Tasks

As healthcare organizations' use of generative AI moves from initial experimentation to scaled deployments, the need to build oversight systems to identify and address ethical challenges assumes greater urgency. Among the AI applications attracting the strongest early interest are clinical summarization tools, which use large language models (LLMs). To assist healthcare organizations weighing adoption of LLMs, we describe an ethical assessment process employed at our healthcare system to identify problems that may affect patient care so problems can be addressed prior to deployment or monitored over time to detect harms. The process uses stakeholder interviewing to explore risks and other concerns arising from integration of AI tools into clinical workflow and identify areas where values and priorities of different stakeholder groups do not align. We describe ethical issues identified in assessments of tools that (1) draft end-of-shift nursing notes and (2) generate clinical notes from conversations between clinicians and patients.