Interactive 3D scene reconstruction for virtual reality based on vision transformer

Three-dimensional scene reconstruction for virtual reality applications demands high geometric accuracy, photorealistic visual quality, and real-time performance. However, existing methods struggle to balance these competing requirements, particularly in complex indoor environments. This paper presents a novel framework for interactive 3D scene reconstruction leveraging Vision Transformers. Our approach employs a hierarchical Vision Transformer backbone to extract multi-scale features with global context modeling capabilities, followed by a cross-view attention mechanism that establishes robust correspondences across multiple viewpoints without relying on hand-crafted cost volumes. To meet VR's real-time requirements, we introduce an interactive refinement module that enables progressive quality improvement in user-selected regions while maintaining smooth frame rates. Comprehensive experiments on the ScanNet dataset demonstrate significant improvements over state-of-the-art methods, achieving 9.2mm Chamfer Distance and 33.5dB PSNR, representing 20.0% and 7.4% improvements respectively compared to the best baseline. Our method maintains real-time performance at 32 FPS on standard VR hardware, making it practical for immersive applications. The proposed framework effectively combines geometric accuracy, visual fidelity, and computational efficiency, advancing the state-of-the-art in VR-based 3D reconstruction.