Multimodal artificial intelligence for human–computer interaction enhanced breast lesion diagnosis

This study aims to enhance breast ultrasound diagnosis by developing a multimodal artificial intelligence (AI) system that integrates lesion detection, diagnostic fusion, and human–computer interaction. We propose KIDNet, a deep learning model for detecting and scoring lesions. A Transformer-based network then fuses ultrasound images, AI-derived scores, and clinical data for final diagnosis. Our results demonstrate that the multimodal fusion approach achieves a diagnostic accuracy of 93.1%, significantly outperforming unimodal and simple fusion methods. Furthermore, we explore different interaction modes for image selection, finding that a human–computer collaborative approach yields the highest accuracy of 95.4%. This work establishes a framework for AI-assisted breast ultrasound that improves workflow efficiency and diagnostic performance through synergistic human-AI collaboration. • KIDNet detects breast lesions efficiently in ultrasound scans. • Transformer fuses imaging, AI scores, and clinical data for diagnosis. • Human-AI collaboration enhances image selection workflow efficiency. • Collaborative AI mode achieves highest diagnostic accuracy (95.4%).