AV2TS: A Multivariate Time Series Modeling Framework for Audio-Visual Segmentation

Audio-visual segmentation (AVS) is a challenging multimodal task that needs to fuse the spatial-temporal audio-visual features to achieve pixel-wise segmentation of sounding objects. This work presents AV2TS, which is a novel spatial-temporal framework for AVS. In contrast to the previous AVS approaches where temporal features are secondarily concerned, we cast AVS as a multivariate time series modeling task in which each frame sequence and its corresponding audio sequence form two time series to highlight the significance of temporal features in AVS. In AV2TS, the perception of each modal sequence is expressed as an intra-series feature extraction process, and the cross-modal fusion task is described as inter-series interactions. Specifically, the intra-series features of the frame and audio sequences are extracted via temporal attention. Moreover, an inter-series cross-modal fusion module is implemented by sharing the shared temporal attention maps of the frame and audio sequences. Additionally, AV2TS contains a one-stream fusion module that unifies the learning and relation modeling processes applied to spatially compact visual features and nontrivial acoustic features. Extensive experiments conducted on three AVSBench datasets demonstrate the state-of-the-art performance of AV2TS in both segmentation and semantic segmentation scenarios. Code is released onhttps://github.com/OrigamiSL/AV2TS.