Multi-scale dual-domain fusion net based on improved transformer for unsupervised spectral reconstruction

Spectral reconstruction (SR) refers to recovery high-fidelity hyperspectral images (HSIs) from an image with limited spectral information. Contemporary methods predominantly rely on supervised learning, necessitating massive paired HSI and RGB images – an assumption often impractical in real-world applications. Furthermore, these approaches mainly focus on spatial domain feature and neglect the high-frequency information, thereby limiting reconstruction fidelity. To conquer these limitations, in this paper, a multi-scale dual-domain fusion net (MDFN) is proposed. Firstly, we introduce a shallow feature extraction (SFE) module to obtain shallow feature. Then, a multi-scale frequency feature extraction (MFFE) module that explicitly captures fine-grained frequency components at multi-resolutions is proposed. Concretely, an improved transformer with a novel spectral-wise attention mechanism is proposed in MFFE module to selectively emphasize informative spectral channels, enhancing the power of frequency feature extraction. Meanwhile, a multi-scale spatial feature extraction (MSFE) module is developed to extract multi-scale spatial features with greater contextual awareness. Next, the dual-domain feature is integrated via a cross domain learning (CDL) module, which enables cross-domain interaction and robust spectral reconstruction. Finally, the fusion feature is reconstructed to HSI via reconstruction mapping (RM) module. What is more, to guide the unsupervised learning process, a novel frequency-aware loss function that integrates traditional pixel-wise constraints with spectral discrepancy metrics is proposed to reinforce frequency-consistent reconstruction. Extensive experiment demonstrates that our MDFN significantly outperforms existing state-of-the-art SR methods without the need for paired training data.