OpenVL: Bridging 2-D and 3-D Worlds for Open-Vocabulary 3-D Scene Understanding

Open-world 3D scene understanding is critical for applications like autonomous driving and robotics, but it faces challenges due to the scarcity of dense 3D-text pairings. While 2D vision-language models offer rich pretraining knowledge, bridging the 2D-3D gap remains challenging. To address this issue, we propose OpenVL and utilize the framework of the Vision-Language model for Open-vocabulary understanding. First, we construct a channel that connects 2D and 3D features through the method of 3D Patches, which largely avoids the problems arising from the scarcity of 3D data. Secondly, we propose a Multi-level 3D Patches Alignment. By achieving dense alignment with 3D Patches at the point, instance, and scene levels, our method not only preserves the spatial perception ability within 3D scenes but also enables a fine-grained understanding of these scenes. Experimental results across multiple bench-marks demonstrate that our approach has achieved state-of-the-art performance. OpenVL achieves significant performance improvements on ScanNet (+6.7% mAP) and nuScenes (+1.9% mAP). More importantly, it demonstrates excellent zero-shot generalization capabilities in downstream tasks including 3DVQA and 3D Embodied Planning.Embodied Planning.