Research Progress on Technologies and Applications of Human-Computer Interaction Gestures Based on Vision and Sensors

Against the backdrop of rapid Artificial Intelligence (AI) and Internet of Thing (IoT) advancement, traditional HCI modalities (keyboards, mice) show limitations, spurring attention to intuitive gesture interaction—a core part of NUIs. This technology mainly has two routes. Vision-based systems use cameras, with deep learning (e.g., YOLOv5 for hand detection, CNN for feature extraction) enabling tasks like keypoint and spatio-temporal feature analysis, and real-time recognition via stereo image fusion. Sensor-based systems include wearable (EMG sensors, smart gloves) and non-wearable (Leap Motion, Kinect, WDHS) devices, with multi-sensor fusion enhancing robustness; bioimpedance measurement is an emerging non-wearable method. These technologies apply to intelligent HCI, disability assistance, VR/AR, and healthcare but face challenges like poor environmental robustness and high computing demands, playing a key role in advancing HCI. Looking forward, gesture recognition is expected to evolve toward more natural, adaptive, and multimodal interaction modes by integrating speech, eye tracking, and emotion recognition for context-aware communication.