Multimodal diagnosis: co-attention transformer for accurate pneumonia classification from chest radiographs and clinical reports

Abstract Chest x-rays are tricky; the fine details and overlaps make it difficult for doctors to diagnose lung issues quickly and accurately. Image-only reading has limitations because features look too similar, and not everyone spots them the same way. Old methods also fall short at blending pictures with reports, so the full story between visuals and words gets lost. The CoATr architecture employs co-attention mechanisms and transformer models to improve cross-modal interactions while boosting multimodal task performance. The system builds a framework that unifies visual chest radiographic data with clinical report semantic information to achieve accurate multi-label classification of chest diseases. The model uses a Swin Transformer backbone to extract detailed visual information from thoracic radiographs. ClinicalBERT was used for clinical report analysis to create contextualised text embeddings. The co-attention mechanism functions to create a shared model which enables visual and textual elements to interact, thus improving their performance through mutual enhancement. The CoATr model was evaluated using the NIH ChestX-ray14 dataset, which contains over 1,12,000 frontal chest radiographs annotated with 14 thoracic disease labels and the associated clinical text. In our tests, the model performed better than standard CNNs and single-mode transformers, and the AUC, precision, and recall all improved noticeably. Since it handles images and text together while explaining its decisions, hospitals can use it to auto-classify chest diseases.