Multimodal Swin Transformer with Graph Neural Networks and Contrastive Learning for Comprehensive Spinal Abnormality Detection

AbstractAccurate detection of spinal abnormalities, such as Spondylolisthesis and Lumbosacral Transitional Vertebrae (LSTV), iscritical for effective diagnosis and treatment planning. However, traditional diagnostic methods are time-consuming and proneto human error, particularly in identifying subtle abnormalities. Existing automated approaches face challenges in handlingclass imbalance, incorporating multimodal data, and providing explainable outputs, which limits their clinical applicability.This study proposes a novel multimodal deep learning framework that integrates Swin Transformer for hierarchical imagefeature extraction, Graph Neural Networks (GNNs) to model spatial relationships between vertebrae, and Contrastive Learningto address class imbalance. The framework combines X-ray images (AP and LA views) with patient metadata, including age,gender, and clinical history, to provide a holistic diagnostic model. The proposed model was evaluated using a publiclyavailable spinal abnormality dataset and outperformed existing methods in classification accuracy, precision, recall, andMatthews Correlation Coefficient (MCC). The model achieved an overall accuracy of 92.7%, with a recall of 91.0% and anAUC-ROC score of 93.8%, demonstrating superior performance compared to baseline methods such as ResNet andEfficientNet. The inclusion of Contrastive Learning and Balanced Batch Sampling significantly improved the detection ofminority classes, such as Left and Right Laterolisthesis.The findings of this study have practical implications for clinicalpractice by offering a robust, interpretable, and scalable framework for spinal abnormality detection. Future research couldexplore integrating additional imaging modalities and refining explainability techniques to further improve the model’sreliability and transparency in real-world clinical applications.This article has not yet been formally published in a journal and is shared here for research dissemination and review purposes. Source code related to this study is available at:https://github.com/aiosmartlab/SpineAI-Multimodal