Temporally-aware differentiable frame compression network for action recognition

Temporal information is crucial for action recognition. However, current deep learning-based frame selection methods often overlook the temporal relationships between frames, resulting in a certain loss of critical temporal information. Moreover, the widespread use of discrete frame sampling mechanisms limits the feasibility of end-to-end training. To address these issues, we propose a temporally-aware differentiable frame compression network. It can model the alignment between frames and their temporal positions by jointly leveraging attention mechanisms and sequence relevance scoring, thereby selecting the frame sequences that contribute the most to the action recognition task. In addition, a differentiable one-hot matrix compression operation is introduced to replace traditional discrete sampling, effectively resolving the gradient blocking problem and enabling end-to-end training. We conducted extensive experiments on UCF101 and HMDB51. The results demonstrate that our method significantly outperforms a number of state-of-the-art approaches.