MDRA: A Motion-guided Dual-stream Recurrent Attention Framework for Dynamic Hand Gesture Recognition

Dynamic Hand Gesture Recognition (DHGR) aims to detect dynamic hand movements by leveraging the features and continuity of video frames. Existing methods mainly utilize backbone networks to extract latent features from individual video frames and sequence modeling through a Transformer. However, the hand usually occupies a relatively small proportion in the video, resulting in a large amount of invalid information in the extracted features, which affects the model’s robustness and subsequent temporal modeling performance. Moreover, the traditional Transformer structure has a high time complexity, which affects the model’s operational efficiency. To address these issues, we propose a novel data preprocessing and data fusion approach. It filters the hand contour using the motion vector of video coding and extracts features from RGB images and contour images through a dual-stream network. Additionally, a Gated-MLP GCN (GM-GCN) fusion module is proposed to fully fuse the dual-stream features. Meanwhile, we developed an Efficient Multi-scale Recurrent Attention (EMRA) module as our temporal modeling network, which adopts a recurrent structure similar to an RNN for attention calculation, enabling efficient parallel training of the model. Moreover, it better captures the details and dynamic changes of gestures through wavelet transform and multi-scale pooling strategies. Extensive experiments demonstrate that our proposed framework achieves highly competitive results on key benchmarks (e.g., 83.87% accuracy on NVGesture) while ensuring computational efficiency, reducing MACs by 28% compared to the standard Transformer.