Interactive Virtual-Physical safety training for manufacturing workshops via vision and LLMs

Effective operational training for production equipment is crucial to ensure personnel safety and realize the human-centric vision of Industry 5.0. However, existing Mixed Reality (MR)-based training methods in manufacturing workshops face two major challenges: difficulty in the semantic perception of equipment and lack of context-aware knowledge responses. To address these challenges, this paper proposes an interactive virtual-physical safety training for manufacturing workshops via vision and Large Language Models (LLMs). First, a methodological framework is established. Next, a machine vision-based semantic segmentation method combined with Vuforia multi-target recognition is employed to enable semantic understanding and 3D virtual-physical mapping of training equipment. Subsequently, a knowledge-grounded question answering method is introduced by integrating knowledge graphs and LLMs to support context-aware training feedback. Finally, a prototype safety training system is developed on HoloLens 2 and experimentally validated. Results show that, compared with traditional paper-based training, the proposed method shows potential to improve trainees’ performance in short-term tasks, particularly in skill-based training tasks under the present experimental conditions. These findings demonstrate its potential as a technical enabler for human-centered industrial safety training for workshop operators.