RSGM: a multi-modal image matching scheme guided by regional semantics

Abstract Multimodal image feature matching technology plays an important role in computer vision research, and a lot of work has been done to improve the matching performance. However, existing matching methods lack sufficient extraction of descriptor information and clear semantic guidance during the matching process. Most descriptors only contain pixel information, neglecting region-based and inter-image information. While some approaches attempt to incorporate semantic information into descriptors, they do not utilize this information during the actual matching stage, leading to potential incorrect matches when similar object types are present in the images. In order to solve these problems, We put forward a new multimodal local image feature matching method that includes a novel descriptor extraction method and a new matching scheme. During the feature extraction and fusion stage, similarity information, regional semantic information of image pairs, as well as neighboring pixel information of corresponding match points from different images are fused to enrich the feature descriptors. In the matching stage, a method guided by regional semantic information(RSGM) is employed to enhance the overall matching effect. Finally, extensive experiments demonstrate that RSGM performs exceptionally well on multimodal image datasets. The code for RSGM is available at: https://github.com/LiaoYun0x0/RSGM.