Attention distillation-enhanced multimodal adaptive fusion for action recognition

A multimodal adaptive action recognition method based on attention distillation is proposed to address irrelevant information interference in multimodal feature extraction and insufficient modal complementarity/consistency in fusion. First, a dual-branch FPN processes RGB and depth map data in parallel for multi-scale feature extraction, while skeletal data undergoes skeleton graph modeling. A feature decoupling module and attention distillation loss function are designed to optimize via loss minimization backpropagation, enhancing task-related feature extraction and reducing noise-induced redundancy. In fusion, a Transformer uses its Encoder-Decoder structure to adaptively adjust multimodal fusion weights. Finally, a spatiotemporal graph convolutional network performs spatiotemporal modeling and classification on enhanced skeleton graph features, capturing human joint spatial correlations and movement temporal dynamics to improve classification accuracy. Experiments on NTU RGB-D60 and NTU RGB-D120 datasets show the method outperforms benchmark algorithms, verifying its effectiveness in action recognition.