Joint Optimization Framework for High-Fidelity Multispectral Filter Array Imaging

Multispectral filter array (MSFA) imaging enables low-cost, real-time, and compact spectral sensing, making it highly suitable for IoT applications such as smartphone imaging, smart agriculture, UAV-based remote sensing, and wearable diagnostics. However, due to the strong coupling among spectral response functions (SRFs), MSFA pattern, and reconstruction algorithms, achieving high-fidelity reconstruction remains a significant challenge. Conventional independent optimization of each module often leads to suboptimal results, highlighting the necessity of a co-optimized design method. To this end, we propose an end-to-end spectral reconstruction framework that jointly optimizes the SRF, MSFA pattern, and reconstruction network. In our method, the SRFs are parameterized as Gaussian functions defined by the center wavelength and full width at half maximum (FWHM). The MSFA pattern is designed using the optimal sphere-packing (OSP) method to enhance sampling uniformity and reduce artifacts. In addition, a dual-stage reconstruction pipeline is introduced: the first stage employs a Res2FFT-based U-Net with multi-scale gated attention (MGA) for spatial demosaicing, and the second stage uses an enhanced deep residual network (EDSR) for spectral super-resolution. All components are jointly trained under a hybrid loss with supervision from both multispectral and hyperspectral reconstruction. Experiments on three benchmark datasets demonstrate that our method achieves superior spatial and spectral fidelity compared to state-of-the-art methods. Furthermore, the model has been successfully deployed on an Nvidia Jetson-based IoT device, achieving real-time inference at 45 ms per frame, validating its potential for real-time spectral imaging in IoT systems.