A hybrid CNN-transformer network for magnetic resonance imaging-guided near-infrared spectral tomography

Magnetic resonance imaging (MRI)-guided near infrared spectral tomography (NIRST) is a noninvasive and promising multimodal technique for early breast cancer detection and precise diagnosis. However, existing image-guided NIRST methods have not fully exploited the complementary information provided by MRI and NIRST, which limits reconstruction quality and diagnostic performance. To address this challenge, we propose a hybrid convolutional neural network-Transformer architecture (CNN-Trans) for MRI-guided NIRST reconstruction. Specifically, CNN-Trans employs convolutional neural networks (CNNs) to extract local features from NIRST measurements, while Transformers are used to capture global contextual information from MRI images. Dynamic feature fusion (DFF) modules are further incorporated to effectively integrate multimodal features, thereby enhancing NIRST reconstruction quality. Numerical simulation results demonstrate that, compared with three state-of-the-art methods, CNN-Trans reduces the average mean squared error (MSE) by at least 3.81%, while increasing the average peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and total hemoglobin (HbT) contrast by at least 1.81%, 7.23%, and 2.92%, respectively. Furthermore, the CNN-Trans trained solely on simulated datasets was directly applied to clinical data from 16 patients without retraining or fine-tuning, and achieved a sensitivity of 90.9%, an accuracy of 87.5%, and an area under the curve (AUC) of 0.909. These results indicate that CNN-Trans holds strong potential for clinical application in breast cancer imaging.