CausalVEE: Adaptive KeyFrame-Based Causal Temporal Modeling for Video Event Extraction

Video event extraction (VEE) aims to identify salient events in videos and accurately extract their associated arguments and semantic roles. Unlike previous methods that primarily focus on modeling visual content, we emphasize the critical importance of capturing causal temporal relationships between events, which are often overlooked in existing research. To address this gap, we propose a CausalVEE framework, comprising three core components. First, we design an Adaptive Keyframe Extraction (AKE) module that employs Gumbel-Softmax sampling to adaptively select keyframes through a differentiable frame-skipping strategy, effectively filtering redundant content in long videos. Second, we introduce an Event Content Extraction (ECE) module to capture rich semantic information of events by modeling the global scene context, object states, and their interactive relationships within the selected keyframes. Finally, we incorporate the Mamba architecture into the VEE task to achieve causal temporal modeling and further enhance it with the Causal Memory-Augmented Mamba (CMA-Mamba) module. This module introduces a lightweight causal attention mechanism as a Support Memory Block (SMB), which alleviates the problem of early information loss in long sequences without compromising the causal modeling principle. Comprehensive experiments on the large-scale VidSitu and VidEvent datasets show that our approach sets a new benchmark in both verb classification and semantic role prediction tasks.