Real-time visual localization with robust spatial perception

Read full text Summarise with VIRAT
Authors: Dongshuo Zhang
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Visual localization is a fundamental capability for autonomous systems, including mobile robots, autonomous vehicles, drones, and AR/VR devices, as it enables accurate estimation of camera pose within a previously observed environment. Reliable localization is essential for downstream tasks such as navigation, mapping, and motion planning. Despite substantial progress, existing visual localization systems continue to face three practical challenges: (1) performance degradation under viewpoint changes and environmental variations, including season, illumination, weather; (2) high computational cost that restricts deployment on resource-constrained platforms; and (3) limited long-term operational ability to incrementally adapt to new environments while preserving performance on previous domains. This thesis addresses these challenges through a unified hierarchical visual localization framework that systematically improves robustness to environmental variations, reduces computational overhead for real-time deployment, and enables long-term adaptation across evolving operational environments. To achieve fast and scalable localization in large-scale environments, we adopt a two-step coarse-to-fine localization paradigm. The first stage performs coarse global localization (visual place recognition) using global descriptors and nearest-neighbor (NN) matching. To enhance robustness against severe environmental variations and viewpoint changes, we propose adaptive training strategies based on Environment-Aware Sampling and Loss (EASaL) and Curricular Contrastive Loss (CCL). EASaL promotes systematic exploitation of informative cross-domain pairs, i.e., images of the same place captured under different illumination, weather, and seasonal conditions. CCL introduces progressive difficulty scheduling based on graded similarity (viewpoint changes), enabling the model to gradually focus on harder viewpoint variations during training. Together, these strategies encourage the global descriptors to learn task-relevant representations that remain stable across drastic environmental and geometric changes. The second stage performs fine localization through local feature extraction and geometric matching. Since highly robust models are often computationally expensive, this part of the thesis focuses on improving efficiency without compromising performance. We develop HybridFeat, a local feature description network based on a hybrid CNN–Transformer architecture. By incorporating an Adaptive Local–Global Feature Fusion (AdaLGDF) module, the network selectively integrates global context into ambiguous local regions while preserving distinctive structures. This design enhances robustness to visual ambiguities (e.g., repetitive patterns and viewpoint changes) while reducing computational cost by 7×–100× in GFLOPs compared to state-of-the-art methods, thereby enabling real-time local matching. To further improve local matching efficiency and robustness, we propose a two-step sparse-to-dense matching strategy. The method first applies a training-free self-attention mechanism to select spatially distributed discriminative patches, followed by a constrained spatial matching step to establish geometrically consistent correspondences. This design substantially reduces redundant similarity comparisons among local descriptors while enhancing robustness to occlusion and extreme viewpoint variations. It achieves up to 210× faster inference than traditional dense matching pipelines while maintaining competitive precision. Building upon the robust and efficient hierarchical framework established in the previous paragraphs, the final part of this thesis addresses the challenge of continual adaptation under sequential domain shifts (e.g., new locations, illumination, weather, and seasonal changes) encountered during long-term deployment. While the existing system excels in static scenarios, it typically lacks the mechanism to evolve without retraining. To bridge this gap, we introduce a parameter-isolation-based continual learning mechanism that enables the framework to adapt to newly observed environments while mitigating catastrophic forgetting. By allocating and selectively routing domain-specific aggregation parameters, the proposed approach performs lightweight updates that preserve previously learned representations while acquiring new knowledge. This effectively mitigates catastrophic forgetting, a common failure mode in traditional adaptation strategies. Extensive experiments demonstrate that our method achieves substantial improvement across sequential domains, whereas conventional regularization and replay strategies exhibit noticeable degradation. This final component transforms the framework from a static localization model into an incrementally adaptable system, realizing the objective of long-term operation in evolving real-world environments. Collectively, the proposed framework advances visual localization toward a robust, efficient, and incrementally adaptable system, providing a systematic solution for reliable long-term deployment in dynamic real-world environments while supporting real-time operation at camera frame rates (30 Hz).

Fields of Study & Subjects

Robotics and Sensor-Based Localization Advanced Image and Video Retrieval Techniques 3D Surveying and Cultural Heritage

Publication details

Journal / source
DR-NTU (Nanyang Technological University)
Publication date
1 January 2026
Language
EN
Document type
Thesis