Human-Interactive Mixed Reality System for Entity Recognition

Reliable perception is essential for collaborative robots operating safely in shared human environments. However, automated entity detection systems still produce errors that degrade a robot's understanding of its surroundings. We present a Human-in-the-Loop (HITL) framework that enables user operators to validate and correct entity recognition and detection through an interactive Mixed Reality (MR) application and interface. Detected entities are visualized as aligned holograms, allowing users to confirm or remove them through intuitive, gesture-based spatial interactions. Our proposed method demonstrates that this shared environment and its interaction approach are functional and effective for correcting detections in real-time. By integrating the HITL approach, our system evaluation produces a more accurate representation of the shared environment and establishes the foundation for future extensions, including safer and more effective human–robot interaction and collaboration.