Multimodal Human–Urban Robot Interaction Interface Using Large Language Models, Computer Vision, and Virtual Reality

Robotic systems are increasingly being developed to enhance intelligence and user interaction within the built environment, supporting low physical effort lifestyles. However, pre-programmed robots often lack the situational awareness to adapt to unstructured domestic environments and result in task errors. Verbal corrections typically lag, often after mistakes have already caused workpiece damage. To address these challenges, we present a real-time user monitoring and proactive plan adaptation system designed to improve task fluency and user engagement. The system integrates an LLM-based commander model, a computer vision (CV)-based engagement recognition module, and a virtual reality (VR) interface for low-workload communication. The LLM acts as a central controller, interpreting inputs, initiating interactions, and translating prompts into robot instructions. Meanwhile, the VR-based emotion monitoring system detects negative user emotions and dynamically adjusts work plans to preemptively avoid errors. Physical embodiment with a desktop setup was conducted to validate and evaluate the system’s efficiency.