A Cross-Modal Retrieval Framework for Radiology Reports and Chest X-Ray Images

Accurate retrieval of relevant radiology reports and images is essential for clinical decision support and large-scale medical data management. This study proposes MedFuse-CLIP, a domain-tuned cross-modal retrieval framework that aligns chest X-ray images with corresponding radiology reports using a dual-encoder OpenCLIP ViT-B/32 backbone. The model introduces two key innovations: adaptive semantic hard-negative mining, which enhances discriminative learning across visually similar pathologies, and a retrieval-aware margin contrastive loss, which stabilizes alignment within the embedding space. Experiments on the MIMIC-CXR-JPG dataset demonstrate strong performance, achieving Recall@10 scores of 87.5% for image→text retrieval and 83.4% for text→image retrieval under fine-tuned cross-modal alignment, along with an average AUROC of 0.879 for zero-shot disease classification across 14 thoracic pathologies. Dimensionality reduction analysis confirmed that compact 256-dimensional embeddings preserve more than 98% of retrieval accuracy while halving storage requirements. The results indicate that MedFuse-CLIP matches or exceeds existing radiology vision-language models such as BioViL and GLoRIA while operating efficiently on a single consumer GPU.