MHPF-HSI: A Multimodal Framework for Hepatocellular Carcinoma Prognosis Prediction Integrating Microscopic Hyperspectral Imaging

Accurate prognosis prediction in hepatocellular carcinoma (HCC) remains a critical challenge. This paper introduces the Multimodal Framework for HCC Prognosis Prediction Integrating Microscopic Hyperspectral Imaging (MHPF-HSI), which leverages hyperspectral images (HSIs), clinical data, and pathology reports from our custom multimodal dataset to significantly enhance prognosis prediction accuracy. Our investigation reveals that no existing open dataset includes prognosis data integrating HSIs, pathology reports, and clinical information. Therefore, we collaborated with multiple hospitals to create this custom dataset. Our novel Multiscale Spectral Attention Network with Spectral Selection innovatively decouples spatial and spectral features at multiple scales, effectively minimizing noise and redundancy to improve data quality. Furthermore, our threshold-free spectral selection method dynamically identifies and extracts crucial spectral information without human intervention, ensuring precise feature utilization. The integration of these features through our Multimodal Residual Feature Fusion Module, coupled with the Balanced Modality Training strategy, robustly addresses overfitting and improves model reliability across modalities. Experimental results demonstrate that our framework enhances prognosis prediction accuracy by 2% to 5% over conventional RGB imaging methods. Extensive validations confirm the MHPF-HSI framework's superior performance, establishing a new benchmark in HCC prognosis.