MERA: Multimodal and Multiscale self-Explanatory model with considerably reduced annotation for lung nodule diagnosis

Background and objective: Lung cancer, a leading cause of cancer-related deaths globally, emphasises the importance of early detection for improving patient outcomes. Pulmonary nodules, often early indicators of lung cancer, necessitate accurate and timely diagnosis. Despite advances in Explainable Artificial Intelligence (XAI), many existing systems struggle to provide clear, comprehensive explanations, especially in scenarios with limited labelled data. This study introduces MERA, a M ultimodal and M ultiscale self- E xplanatory model designed for lung nodule diagnosis with considerably R educed A nnotation requirements. Methods: MERA integrates unsupervised and weakly supervised learning strategies, including self-supervised learning techniques and Vision Transformer architecture for unsupervised feature extraction, followed by a hierarchical prediction mechanism that leverages sparse annotations through semi-supervised active learning in the learned latent space. MERA explains its decisions on multiple levels: model-level global explanations through semantic latent space clustering, instance-level case-based explanations providing similar instances, local visual explanations via attention maps, and concept explanations based on critical nodule attributes. Results: Evaluations on the public LIDC dataset underscore MERA’s superior diagnostic accuracy and self-explainability. With only 1% of annotated samples, MERA achieves diagnostic accuracy comparable to or exceeding that of state-of-the-art methods that require full annotation. The model’s inherent design delivers comprehensive, robust, multilevel explanations that align closely with clinical practices, thereby enhancing the trustworthiness and transparency of the diagnostic process. Conclusions: MERA addresses critical gaps in XAI and lung nodule diagnosis by providing a self-explanatory framework with multimodal and multiscale explanations and significantly reduced annotation needs. The demonstrated viability of unsupervised and weakly supervised learning in this context lowers the barrier to deploying diagnostic AI systems in broader medical domains. MERA represents a significant step towards more transparent, understandable, and trustworthy AI systems in healthcare. Our complete code is open-source available: https://github.com/diku-dk/credanno .