Semantic-Enhanced Scene Coordinate Regression Network for Visual Localization

Visual localization is a critical component in Geographic Information Systems (GIS) and Location-based Services (LBS), as it enables precise spatial awareness and navigation. Though current learning-based scene coordinate regression methods have achieved superior accuracy in indoor environments, using the entire or the uniformly sampled feature map for training the scene coordinate regression model often leads to less stable localization performance due to the outliers or low-texture areas, posing challenges in adapting to outdoor environments. To tackle this issue, we propose semantic-enhanced scene coordinate regression network (SEN), which improves visual localization capabilities, particularly in outdoor environments. The SEN integrates an auxiliary semantic feature extraction branch to actively determine the reliable region for scene coordinate regression, filtering out the non-informative pixels. Subsequently, the network creates a randomized buffer to regress the scene coordinates, effectively reducing correlated gradients and achieving efficient optimization. The final regression layer accurately maps 2D pixel coordinates to their corresponding 3D scene coordinates, and the pose of the camera is obtained through differentiable RANSAC. Extensive experiments on three representative datasets demonstrate that SEN achieves state-of-the-art performance across various scenes, delivering exceptional performance in both indoor and outdoor environments.