Modeling Semantic and Localization Uncertainty for Weakly Supervised Temporal Action Localization

The objective of weakly supervised temporal action localization (WTAL) is to accurately identify the temporal intervals of actions using only video-level annotations for training. Existing cross-modal WTAL methods integrate vision-language models to provide rich semantic supervision, aiming to alleviate the inherent supervision limitations in weakly supervised scenarios. However, it is crucial to acknowledge that contemporary cross-modal methods incorporate textual information simultaneously, which inevitably introduces uncertainty in the alignment of cross-modal semantics. Moreover, previous approaches typically output deterministic temporal localization results, while neglecting to evaluate the predictive uncertainty and confidence of localization results. To address the above issues, we propose a novel Modeling Semantic and Localization Uncertainty (MSLU) framework for WTAL, which can simultaneously model the semantic uncertainty in cross-modal representations and the uncertainty of localization results to achieve more precise and robust temporal action localization. Specifically, we propose the Probabilistic Semantic Uncertainty Modeling (PSUM) module, which utilizes probabilistic encoding to capture diverse cross-modal feature representations, effectively mitigating semantic ambiguity in feature alignment. In addition, we propose the Uncertainty-guided Localization Estimation (ULE) module, which leverages evidential deep learning to estimate predictive uncertainty of localization results in weakly supervised scenarios. Through extensive experiments on benchmark datasets including THUMOS14, ActivityNet1.2, ActivityNet1.3, and FineAction, our framework demonstrates superior performance compared to existing state-of-the-art methods. The empirical results validate the effectiveness of simultaneously modeling both semantic and localization uncertainty.