Attention guided multimodal learning for classification of oral lesions using histopathology and clinical data

Oral cancer and leukoplakia are significant conditions with complex diagnostic challenges. Histopathological images provide essential visual cues for identifying abnormalities and clinical data offers crucial context such as lifestyle factors and patient demographics, both of which are vital for accurate diagnosis. This paper proposes a multimodal deep learning framework which integrates histopathological image data and corresponding clinical features to improve the classification of oral lesions. The framework employs modality-specific pipelines, leveraging a cross-modal attention mechanism for data fusion The model integrates image features from a convolutional neural network with clinical features processed through a multi-layer perceptron, optimizing prediction accuracy. Evaluated on a public dataset, the proposed model demonstrated an accuracy of 97.2% and F1-score of 96.9%, outperforming both traditional machine learning methods and single-modality deep learning models. This approach, which integrates visual and clinical data, offers a robust and accurate diagnosis for early detection. It is also able to assist in differentiation of malignant and potentially malignant oral lesions. Future work can include enhancing model generalizability with multi-institutional data and incorporating additional diagnostic modalities for broader clinical applicability.