Fusion of Infrared and Visible Images Based on Iterative Dual-Branch Attention and Modality Discrepancy Guidance

The fusion of infrared and visible images aims to generate images that provide a more comprehensive description of the scene. Convolutional neural network and transformer are two commonly used methods in image fusion. The former focuses on extracting local features but lacks a global receptive field, and the latter can extract global information but ignores the discrepancy information between modalities. In view of this, we propose an efficient fusion network based on iterative dual-branch attention and modality discrepancy guidance (IDMDNet), which consists of three components. In the first component, we use shallow and deep feature extraction modules to extract features. In the second component, we first design an iterative dual-branch attention module to capture important features of each modality, thereby preserving important information. Secondly, to model the global information of the source image, we introduce a transformer module to achieve the fusion of shallow global information. Further, we design a modality discrepancy guided fusion module to promote the fusion of modality discrepancy information and ultimately achieve modality information complementation. In the third component, we introduce an invertible neural network to reduce the loss of feature information, thus achieving high quality image reconstruction. Finally, we construct a loss function for IDMDNet that includes intensity, gradient, and multiscale structure to motivate the network to preserve texture and target details. Experiments on infrared and visible image fusion benchmark datasets show that the proposed IDMDNet has competitive fusion performance. Remarkably, it can be effectively applied to other infrared and visible datasets without the need for fine-tuning, highlighting its good generalization ability. The source code of IDMDNet has been released athttps://github.com/AHUT-MILAGroup/IDMDNet.