Multimodal fusions for defect detection of photovoltaic panels by mask R-CNN and hawkfish optimization algorithm

Accurate detection of photovoltaic (PV) module defects remains challenging due to environmental variability and the limited fault visibility of single-modality imaging. While RGB and electroluminescence (EL) images provide structural and subsurface information, they fail to capture thermal fault characteristics associated with hotspots, cell mismatch, and localized heating. Integrating infrared (IR) imagery offers complementary thermal cues that are critical for comprehensive PV inspection. This paper proposes a multimodal PV defect segmentation framework based on a modified Mask R-CNN architecture that fuses RGB, IR, and EL modalities at the feature level. A dedicated alignment pipeline combining homography transformation and enhanced correlation coefficient refinement ensures geometric consistency across modalities. A Fusion Attention Block adaptively weights modality-specific features, enabling effective cross-modal representation learning. Model hyperparameters and fusion weights are automatically optimized using the HawkFish Optimization Algorithm to improve convergence stability and segmentation robustness. Experiments conducted on statistically paired RGB–EL–IR datasets demonstrate that incorporating IR imagery significantly improves the detection of thermally driven defects and reduces false negatives in low-contrast and ambiguous regions. The proposed framework consistently outperforms unimodal and bimodal baselines, achieving state-of-the-art segmentation accuracy and enhanced defect localization, particularly for heat-related fault patterns. The results confirm that thermal information provides critical diagnostic value that cannot be recovered from RGB or EL data alone. The adaptive fusion strategy and optimization-driven tuning further enhance generalization under real-world conditions. These findings highlight the importance of IR-integrated multimodal learning for reliable and scalable PV module inspection systems.