ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localization

Dense audio-visual event localization (DAVE) aims to identify event categories and locate the temporal boundaries in untrimmed videos. For such challenging task settings, most studies only employ audio-visual event semantic constraints on the final outputs, lacking progressive cross-modal semantic bridging in intermediate layers. This causes semantic gaps that hinder alignment between representations in audio and visual features, making it difficult to distinguish between event-related and irrelevant background content. Moreover, they rarely consider the correlations between events, which limits the model to infer co-occurring events among complex scenarios. In this paper, we incorporate multi-stage semantic guidance and multi-event relationship modeling, which respectively enable progressive semantic understanding of audio-visual events and adaptive extraction of event dependencies, thereby better focusing on event-related information. Specifically, our event-aware semantic guided network (ESG-Net) includes a early semantic interaction (ESI) module and a mixture of dependency experts (MoDE) module. ESI applys multi-stage semantic guidance to explicitly constrain the model in learning semantic information through multi-stage feature fusion and several classification loss functions, ensuring multi-stage understanding of event-related content. MoDE promotes the extraction of multi-event dependencies through multiple serial mixture of experts with adaptive weight allocation. Extensive experiments demonstrate that our method significantly surpasses the state-of-the-art methods, while greatly reducing parameters and computational load. Our code will be released on https://github.com/uchiha99999/ESG-Net.