141 Multimodal AI framework: End-to-end diagnostic pipeline enhanced with retrieval augmented generation

Background In healthcare, most existing uses of artificial intelligence (AI) focus on ‘narrow AI’. These systems are trained to provide binary outputs (i.e. yes/no) as to whether a finding is present or absent on an image. While useful, the real benefit for patients comes when AI is used not just for problem detection, but also to guide and shape the care pathway for patients with evidence-based management and treatment decisions. In this work, we illustrate a pipeline using a common example of bone fractures in children. Non-specialist healthcare practitioners (HCP) may miss up to 1 in 10 fractures on children’s X-rays. Commercial AI tools for this population are limited, and none currently provide information other than presence/absence of a fracture. Our study presents a multimodal AI framework that takes this further. It combines both image-based fracture detection (i.e. a computer vision algorithm) with text-based guidance drawn from national fracture management guidelines. The goal is not only to assist in fracture detection, but also to support frontline practitioners in deciding how best to manage it.Method We trained an AI model (an object detection model) using a multi-centric dataset of paediatric X-rays to detect the location and type of paediatric-specific fractures. The model’s output is then passed into a retrieval system (a RAG pipeline component) that searches clinical guidance literature to identify the optimal patient management. A concise summary of the recommended care plan is then generated.Conclusion For real transformational benefit and ease, AI systems must demonstrate value beyond narrow tasks. Our study illustrates one such approach – combining image-based fracture detection with guideline-based management advice. Although we focus on paediatric fractures, this framework can be adapted to other clinical tasks.Acknowledgements for funding or support CP is funded by the Great Ormond Street Hospital Children’s Charity (GOSHCC) (Award Number: VS0618)OJA is funded by an NIHR Career Development Fellowship (NIHR–CDF–2017–10–037)SCS is funded by an NIHR Advanced Fellowship Award (NIHR–301322)