XPoint: A Self-Supervised Visual-State-Space-Based Architecture for Multispectral Image Registration

Accurate multispectral image matching presents significant challenges due to non-linear intensity variations across spectral modalities, extreme viewpoint changes, and the scarcity of labeled datasets. Current state-of-the-art methods are typically specialized for a single spectral difference, such as visible-infrared, and struggle to adapt to other modalities due to their reliance on expensive supervision, such as depth maps or camera poses. To address the need for rapid adaptation across modalities, we introduce XPoint, a self-supervised, modular image-matching framework designed for adaptive training and fine-tuning on aligned multispectral datasets, allowing users to customize key components based on their specific tasks. XPoint employs modularity and self-supervision to allow for the adjustment of elements such as the base detector, which generates pseudo-ground truth keypoints invariant to viewpoint and spectrum variations. The framework integrates a VMamba encoder, pre-trained on segmentation tasks, for robust feature extraction, and includes three joint decoder heads: two are dedicated to interest point and descriptor extraction; and a task-specific homography regression head imposes geometric constraints for superior performance in tasks like image registration. This flexible architecture enables quick adaptation to a wide range of sensing modalities. We demonstrate this by training on OPT–TH (optical–thermal) data and fine-tuning on several multispectral settings, including VIS–NIR (visual–near infrared, 0.75–1.4 μm), VIS–IR (visual–infrared, corresponding to long-wave infrared, 8–15 μm), and VIS–SAR (visual–synthetic aperture radar). Experimental results show that XPoint consistently outperforms or matches state-of-the-art methods in feature matching and image registration tasks across five distinct multispectral datasets. Our source code is publicly available at https://github.com/canyagmur/XPoint.