Semantic Modulated Prompting for Few-Shot Audio-Visual Classification

Few-Shot Audio-Visual Classification (FS-AVC) trains models using a limited number of labeled audio and visual sample pairs to capture the classification capability. Deep learning-based audio-visual learning methods often construct complicated frameworks with numerous parameters trained on large labeled datasets, rendering them impractical for FSAVC. The key challenges for FS-AVC are model overfitting, multimodal fusion under temporally asynchronous scenarios, and modality imbalance. Toward these challenges, we propose a novel method called Semantic Modulated Prompting (SMP) to improve the learning process of FS-AVC. This framework implants text as prompting tokens via two components: Promptrefined Audio-Visual efficient Learner (P-AVeL) and Prompttuned Prototypical Regularization (P-PR). By integrating semantic prompts, adapter-based P-AVeLs conduct the prompt-guided latent attention to alleviate the overfitting and achieve effective alignment and fusion. Concurrently, P-PR, the first rebalancing method designed for few-shot scenarios, uses these semantic prompts to accurately evaluate and dynamically adjust the imbalance of two modalities. Extensive experiments demonstrate that the SMP framework consistently outperforms state-of-theart multimodal methods by a large margin. The codes are available athttps://github.com/DennisHgj/SMP\_FSAVC.