Fourier Multispectral Transformer for Robust Hyperspectral Reconstruction and Remote Sensing Segmentation

Abstract Hyperspectral imaging is valuable for computer vision tasks because it captures detailed spatial and spectral data, which are crucial for applications like semantic segmentation, material identification, and aerial imaging. However, its expensive and complex hardware limits its use in common devices such as food scanners, aerial imaging systems, and medical devices. Recently, computational spectral imaging methods have been developed to reconstruct hyperspectral data directly from standard red-green-blue images, enabling spectral reconstruction without costly spectral cameras while maintaining spectral resolution and quality. Despite these advances, reconstructing hyperspectral information from red-green-blue images remains an ill-posed inverse problem. This work introduces the Fourier multispectral transformer, designed for precise spectral recovery and robust performance in real segmentation scenarios. Instead of relying on computationally intensive multihead self-attention, Fourier multispectral transformer employs fast Fourier transform operations for efficient global spectral and spatial mixing, reducing complexity. A lightweight spectral refinement layer further improves wavelength consistency. Our training approach jointly improves reconstruction accuracy and robustness, ensuring that the spectral cubes support a range of vision tasks. We evaluate Fourier multispectral transformer on the ARAD-1K spectral reconstruction benchmark and assess its hyperspectral outputs within a U-Net segmentation framework using the AeroRIT [6] aerial dataset. Additional evaluations on other semantic segmentation and skin classification tasks demonstrate their improved performance. Fourier multispectral transformer surpasses convolutional neural networks and transformer baselines in reconstruction metrics (MRAE, RMSE, PSNR) and attains higher segmentation and classification accuracy, narrowing the gap with models trained on real multispectral data. It adapts well to diverse aerial red-green-blue scenes, providing a practical way to generate hyperspectral-like data for downstream applications without specialized equipment. This research shows that reconstructing spectral reflectance from red-green-blue images offers a reliable, fast, and cost-effective tool for monitoring and analysis across many computer vision tasks.