Development of a Mixed Reality Robotics Testing Environment

Development of a Mixed Reality Robotics Testing Environment Haneen Hanoon This research presents the development of a Mixed Reality (MR) robotic framework that integrates real and virtual environments for experimental robotics, education, and visualization. The proposed system combines a motion-capture system, a simulator, and a projector to create a spatially aligned, interactive workspace where physical robots coexist with projected virtual objects. The framework enables real-robot motion to be tracked in real time and mirrored in the simulated environment, allowing projected obstacles, goals, and paths to be precisely aligned with the physical world. A coordinate transformation and scaling process was implemented to ensure accurate mapping between the motion capture system and the simulation coordinate frames. Software programs were developed to manage real-time pose streaming, calibration, and dynamic visualization. The system was validated through multiple navigation and obstacle-avoidance experiments conducted in both static and dynamic environments, demonstrating the framework’s ability to maintain spatial consistency between virtual and physical spaces. Experimental results demonstrated accurate spatial alignment between the physical and virtual domains and stable real-time performance during testing. For example, the physical robot successfully avoided both static and moving virtual obstacles while following replanned paths displayed on the lab floor using the projector. This work establishes a robust MR foundation for real-time robot navigation, spatial calibration, and educational visualization, bridging physical experimentation and virtual simulation.