Gesture and Sign Language Recognition of the Flexible Capacitive Sensor Based on Deep Learning Models

With the rapid development of artificial intelligence, gesture recognition based on flexible pressure sensors has become a hot topic in human–computer interaction. In this paper, a capacitive flexible pressure sensor with simple fabrication, low cost, and great efficiency is proposed to identify different gestures. The sensor is composed of three main layers: the upper electrode layer, the dielectric layer and the lower electrode layer. The dielectric layer is consist of a step microstructure made of elastic material (Dragon Skin FX-Pro) and carbon black (CB). The upper and lower electrode layers are composed of fabric electrodes. The Dragon Skin FX-Pro/CB sensor has high sensitivity (3.97% kPa –1 ), fast response time and excellent stability. In the experiment, five sensors were fixed at the palm center of the left-hand, the inner fingertip of the right thumb, the side of the right-hand near the little thumb, the second joint of the back of the right middle finger and the inner fingertip of the index finger respectively with a pair of insulating gloves. The capacitance time series signal data from the five Dragon Skin FX-Pro/CB sensors for 9 emotional gestures and 9 sign language gestures were collected. Then, the convolutional neural network (CNN)-transformer deep learning model with excellent feature extraction and data fusion ability was constructed. Finally, the 9 emotional gestures and the 9 sign language gestures are recognized by the CNN-transformer model with great accuracy of 98.06% and 99.44% respectively, which means that the capacitive sensor is highly suitable for capturing different gesture signals, and the CNN-transformer model has wonderful performance in recognizing gestures with different features. All the experimental results demonstrate that the combination of Dragon Skin FX-Pro/CB capacitive sensor, wearable devices, and CNN-transformer model has great application prospects in human–machine interaction, especially in the gesture recognition field.