An Adaptive Multimodal Framework for Designing Intelligent Virtual Agents in Mixed Reality Using Scene Understanding

This research presents a novel design and implementation of an Intelligent Virtual Agent (IVA) in mixed reality that has advanced speech capabilities from large language models and integrates computer vision to perceive the user’s environment and actions in the real-world context. Scene understanding allows the IVA to navigate in the user’s physical space, demonstrate an understanding of the user’s actions, and dynamically interact with real-world entities. We propose a comprehensive framework for this multimodal integration, which enables the IVA to tailor its assistance and provide adaptive guidance to the users based on the actions they take in the real-world. To demonstrate and evaluate the proposed framework, we implemented two novel scenarios. Results demonstrated that participants consistently reported higher engagement, interactivity, and effectiveness with the IVA despite taking more time to complete the task. Moreover, all participants valued the IVA’s ability to adapt to their actions, offering a more personalized experience. Application of a previously introduced multimodal Intelligent Virtual Agent framework in mixed reality across multiple task domainsDesign and implementation of two novel domain-specific scenarios to demonstrate the applicability of the IVA.Comprehensive user studies comparing the IVA with baseline systems across both scenariosDemonstration of real-world IVA capabilities, including navigation, scene understanding, and natural multimodal interaction enabled through spatial mapping, vision, and large language models Application of a previously introduced multimodal Intelligent Virtual Agent framework in mixed reality across multiple task domains Design and implementation of two novel domain-specific scenarios to demonstrate the applicability of the IVA. Comprehensive user studies comparing the IVA with baseline systems across both scenarios Demonstration of real-world IVA capabilities, including navigation, scene understanding, and natural multimodal interaction enabled through spatial mapping, vision, and large language models