GCM-Pose: Generalizable 6D Object Pose Estimation Based on Cross-Modal Feature Matching

6D pose estimation is a crucial subfield of visual measurement, extensively applied in robotics, autonomous driving, and augmented reality. Existing instance-level and category-level 6D pose estimation methods typically rely on prior object models or category labels, limiting their generalization and scalability in complex and dynamic environments. To address this limitation, we propose GCM-Pose, a general 6D pose estimation framework that performs inference using object-specific reference data. Specifically, a sparse Structure-from-Motion (SfM) model is first constructed from a set of RGB images or a short video. Then, GCM-Pose estimates the object’s 6D pose by establishing 2D-3D correspondences between the query image and the SfM model. This approach can be applied to novel objects without any CAD models. To handle cross-modal discrepancies and multi-scale variations in the 2D–3D feature matching process, we introduce a Cross-Modal Multi-Scale Deformable Attention (CMMDA) network trained with a triplet loss learning strategy. CMMDA addresses these issues by aligning feature descriptors from different modalities within a multi-scale latent high-dimensional space, enabling the establishment of precise and robust 2D-3D correspondences. We also construct a large-scale dataset consisting of 200 objects under various real-world conditions, including lighting variation, occlusion, and transparent objects. Experimental results on LineMOD, OnePose-LowTexture, and real-world scenes demonstrate that GCM-Pose achieves SOTA performance, highlighting its effectiveness and wide applicability in 6D pose estimation tasks.