Spatiotemporal perception and motion similarity learning for real-time assembly verification

The increasing demand for personalized products and operational flexibility in Industry 4.0 has exposed critical limitations in traditional quality assurance systems, particularly within human-in-the-loop manual assembly. To address these challenges, this paper introduces an integrated vision-based framework that combines spatiotemporal object detection and motion similarity assessment for the automated verification of procedural correctness. A multi-modal acquisition system, utilizing synchronized RGB and depth data, is employed to support precise 3D hand keypoint reconstruction. For process segmentation, STM-YOLO (Spatiotemporal Multi-View You Only Look Once) was developed. By reconfiguring standard detection heads into stage-transition classifiers, this model achieves an average accuracy of 87%, outperforming the single-view, single-frame baseline (S-S) by an absolute margin of 27.67%, and further validates the complementary contributions of both temporal modeling and multi-view perception modules to performance improvement. To assess operator action conformity without the need for gesture-level annotation, the Siamese Transformer Encoder Network (STEN) is introduced, evaluating 3D hand motion trajectories against golden-standard references. Validated on the filtered FPHA (First-Person Hand Action) benchmark, STEN attains 99.28% accuracy and an F1-score of 0.9858, surpassing classical approaches. The proposed system operates at over 10 fps on laboratory hardware and demonstrates robust generalization across diverse operators and tasks, offering a scalable solution for procedural verification in flexible manufacturing environments.