Nontrivial Audio-Visual Segmentation

In mathematics, the adjective 'trivial' is frequently used to refer to an object with a simple structure. Although it is counterintuitive to imagine that a multimodal task such as audio-visual segmentation (AVS) can be trivial, the trivial acoustic features possessed by the existing AVS models tremendously limit their performance. Indeed, acoustic features are normally averaged and repeated in the time-frequency domain to fit the sizes of visual features used in previous approaches. Thus, the triviality of these acoustic features degrades the effectiveness of audio-visual fusion and segmentation. In contrast, a novel global temporal audio-visual mixer (GTAVM) is proposed in this work; this approach fuses globally the temporal dependencies of audio and visual features along the input video sequence, transforming the average of the acoustic features into the average of the global temporal relations. Therefore, the acoustic features are able to maintain their sizes and generate nontrivial audio queries for segmentation purposes. To further enrich the acoustic features, a hierarchical audio backbone, in which multiple time-frequency resolutions are utilized to generate hierarchical Mel-spectrograms as audio inputs, is proposed in this work. Extensive experiments conducted on three AVS benchmarks demonstrate the state-of-the-art performance of the proposed methods. The source code will be released soon.