Feature Hallucination for Self-supervised Action Recognition

Understanding human actions in videos requires robust integration of multimodal cues beyond raw pixels. This work introduces a deep self-supervised action recognition framework that jointly predicts action concepts and auxiliary features from RGB video, then hallucinates missing modalities at test time to improve recognition without added runtime cost. Two new domain-specific descriptors, Object Detection Features (ODF) and Saliency Detection Features (SDF), are proposed to capture spatial context and motion saliency, integrating them with other modalities such as optical flow, skeleton, audio, and improved dense trajectories. The framework incorporates aleatoric uncertainty modeling to handle noisy or unreliable features, along with a robust loss for stable multimodal fusion. Compatible with popular architectures including I3D, AssembleNet, Video Transformer Network, VideoMAE V2, and InternVideo2, the approach achieves state-of-the-art results on Kinetics-400, Kinetics-600, and Something-Something V2.