Multimodal fault detection model for photovoltaic modules

With the ongoing global transition towards clean energy, the photovoltaic industry has rapidly entered a new stage of large-scale development. To overcome the limitations of single-modality image-based photovoltaic module fault detection models, this study proposes Photovoltaic-DETR, a multimodal fault detection model based on RT-DETR. The model is capable of efficiently processing infrared hotspot images, infrared images, and visible light images of photovoltaic modules. First, a lightweight backbone network is constructed using self-designed ORPELAN and ReLA Block modules, incorporating an auxiliary reversible branch to efficiently extract spatial features of photovoltaic modules. Secondly, a reconstructed feature fusion network is proposed, which integrates an attention-scale sequence fusion mechanism with a reparameterization method to reduce channel concatenation redundancy. Lastly, dynamic upsampling and downsampling are achieved using the DySample module during feature fusion, enhancing the model's perception ability. Experimental results on the UAV-captured photovoltaic module hotspot fault detection dataset, the public infrared photovoltaic module dataset (GB_HSP_modified, PV_Train_Val_28_12), and a self-made visible light dataset show that, compared to the RT-DETR model, the Photovoltaic-DETR model improves mAP@50% by 2.9, 4.9, 2.6, and 5.1% points, respectively. The model's parameter count is reduced by 28.6%, and its computational load is decreased by 28.5%. These results fully demonstrate the excellent adaptability of Photovoltaic-DETR in multimodal fault detection for photovoltaic modules, providing a solid technical foundation for industrial multimodal photovoltaic module fault detection.