Online dynamic task planning with environment observation for human–robot collaboration

Human–robot collaboration has become essential for leveraging the complementary strengths of humans and robots to split their duties. Real-time dynamic task planning that balances task completion time and human fatigue is essential for improving system efficiency and flexibility. However, most existing studies focus primarily on local task feasibility, neglecting system-level optimization in terms of time efficiency, human fatigue, and personalized collaboration adaptability. To address these limitations and enhance adaptability in dynamic environments, this study proposes an innovative real-time dynamic task planning framework integrated with environment observation for human–robot collaboration. On the environment observation side, a lightweight recognition and prediction network with only 10 M parameters is developed, providing rapid and accurate human behavior inference. On the dynamic task planning side, a multi-objective, fast-search strategy that utilizes environment observation outputs is proposed to synchronize robot actions effectively with human dynamically in real-time. A simulated industrial production line involving object handling from shelves to workbench is built, and a dataset of over one million frames capturing human operations is collected. Experimental results indicate the proposed environment observation architecture can rapidly identify human behaviors within 0.025 s including recognizing and predicting human actions and target objects, along with predicting human long-term (3.0 s) motion. The system achieves high accuracies in action recognition (99.4%), action prediction (97.4%), target object identification (96.7%), target object prediction (98.1%), and the motion predictor achieves the lowest MPJPE at horizons from 1.5 to 3.0 s on both PPAH and Human3.6M. Additionally, online dynamic task planning can adapt robot actions according to human dynamics with computation consistently below 0.020 s in our experiments. These performance underscore the novelty and practical application potential of our proposed method in industrial human–robot collaboration.