A Hybrid Multimodal System Combining Vision and Flexible Strain Sensors for Gesture Recognition

This study presents a multimodal gesture recognition system based on the fusion of visual and strain sensor data, aiming to overcome the limitations of traditional gesture recognition methods in complex environments. A flexible strain sensor made from a CNTs-Graphene-TPU composite material was developed to create a smart glove with high sensitivity and stability, capable of accurately capturing finger flexion signals. Meanwhile, by integrating deep learning algorithms, a hybrid gesture recognition framework was constructed that combines visual and strain sensor data. Experimental results show that the system maintains high recognition accuracy (96.31%) across varying lighting conditions, significantly improving gesture recognition accuracy and system robustness. The system shows promising applications, particularly for gesture interaction and sign language translation in challenging lighting environments.