Hybrid Debiasing Transformer With Adaptive Regularization for Video Moment Localization

Video Moment Localization (VML) is a task that seeks to pinpoint the most pertinent segment within an untrimmed video using a linguistic query. Previous works expose the severe data bias issues in VML and note that models avoid understanding visual-textual content by adapting the timestamp distribution. The work investigates data biases from both intrinsic and extrinsic perspectives: The former arises primarily from moment boundary ambiguity and the inputoutput information imbalance. The latter is attributed to the longtail distribution and the semantic bias resulting from the limited tail samples. To reduce the issues, we develop a hybrid multimodal debiasing network with a temporal consistency constraint for VML. Firstly, we propose a multi-temporal Transformer to alleviate boundary ambiguity by merging frame-wise features into segment-wise representations and dynamically aligning with moment boundaries. Subsequently, we implement a temporal consistency constraint to accentuate action information from complex moment context and mitigate the intrinsic bias caused by information imbalance. Moreover, we develop a hybrid linguistic activation module to mitigate the long-tail bias, which offers prior guidance to emphasize distinguishing clues from tail samples. Additionally, we introduce the prior-guided Transformer to alleviate the semantic bias by learning the global semantics of sentences, thereby circumventing the tail-sample overfitting issue. Comprehensive experiments demonstrate the efficacy of our proposed method across three datasets.