Efficient Human Feature Refinement for Weakly Supervised Group Activity Recognition

Weakly supervised group activity recognition (WSGAR) aims to identify the joint activity of a group of people without relying on hand-annotated human bounding boxes. Existing WSGAR methods typically acquire coarse human-level features by pooling from detected bounding boxes or applying human queries with cross attentions. These approaches focus on learning human relations from the acquired features. However, discriminative person-specific clues might be confused with irrelevant backgrounds, hindering the effectiveness of downstream human relation learning. To address this limitation, we propose a Human Feature Refinement framework that enhances human-level information with graph convolutional networks and self-attention. We define in-box regions as tokens and learn their spatial correspondence through GCN and self-attention. By explicitly extracting in-box details and suppressing irrelevant regions, our method acquires more discriminative human-level features for relation learning and group activity prediction. We further propose a Graph-based Token Merging algorithm to reduce the computation cost of Human Feature Refinement, while minimizing information loss and overfitting risk. Experiments show that our method outperforms previous WSGAR methods on Volleyball, NBA and JRDB-PAR benchmarks, with reduced computation cost.