PAB: Points as a Bridge for Omni-Supervised Temporal Action Detection

Omni-supervised temporal action detection aims to leverage various forms of supervised data to improve the generalization ability of action detectors. Existing methods rely on low-quality random proposals for classification and regression, resulting in suboptimal detection performance. To address this, we proposePointsAs aBridge (PAB), a unified pseudo-label generator that integrates unlabeled, fully labeled, and weakly labeled annotations (e.g., video-level and point-level labels), generating high-quality pseudo-labels and improving detection performance. PAB consists of two main modules:Multi-ModalPointMining (MMPM) andPoint-EnhancedTransformer (PET). The MMPM module uses a learnable visual prototype and a frozen textual prototype for each category. These prototypes jointly identify temporal points in the input video where action instances are likely to occur, with each point representing its temporal position and associated action category. Based on these mined points, the PET models a series of explicit action queries to predict the temporal boundaries of the action instance for each point. In this framework, points serve as a bridge, transmitting temporal positions and action categories between the two modules. Based on MMPM's predicted action categories and PET's estimated action boundaries for each point, PAB generates high-quality pseudo-labels for unlabeled and weakly labeled data, enabling a fully supervised detector to achieve state-of-the-art performance in semi- and omni-supervised settings.