A Survey on Large Multimodal Models for 3D Vision and Scene Understanding

The landscape of artificial intelligence has been profoundly transformed by the advent of Large Multimodal Models (LMMs), which integrate the superior language comprehension and generation capabilities of Large Language Models (LLMs) with rich information from other modalities such as vision, audio, and even 3D data. This convergence marks a significant leap towards more holistic and human-like AI systems. In the domain of 3D vision and scene understanding, LMMs are particularly impactful, addressing the inherent complexities of 3D data-including its diverse representations, sparsity, and intricate geometric and semantic structures. Traditional 3D vision methods often struggle with semantic reasoning and open-ended understanding, limitations that LMMs are poised to overcome by leveraging cross-modal contextualization and grounding. This survey provides a comprehensive overview of the rapidly evolving field of LMMs for 3D vision and scene understanding. We delve into their foundational architectural paradigms, core 3D perception tasks enhanced by multimodal inputs, and their contributions to higher-level 3D scene understanding and reasoning, including embodied AI applications. Furthermore, we meticulously review the crucial benchmarks, datasets, and evaluation metrics that drive progress in this field. Concluding with a critical analysis of current challenges, promising future research directions, and essential ethical considerations, this paper aims to serve as a valuable resource for researchers and practitioners navigating the exciting opportunities and complexities at the intersection of LMMs and 3D intelligence.