Text-Centric Sparse Interaction Fusion Network With a Modality Calibrating Module for Multimodal Sentiment Analysis

Despite significant progress in Multimodal Sentiment Analysis (MSA), particularly with methods based on Transformers and Attentions, three major issues remain unresolved. First, existing unimodal representation learning methods overlook the noise and redundancy present within unimodal sequences. Second, current methods ignore the uneven distribution of emotional information across different modalities and fail to consider the highest contribution of text. Third, they neglect the issue of inter-modal noise redundancy in modal interactions. To address the aforementioned issues, this paper proposes a Text-Centric Sparse Interaction Fusion Network with a Modality Calibrating Module (TC-SIFN-MCM). A Modality Calibrating Module (MCM) is introduced during the unimodal representation learning phase to suppress noise within modalities. A Text-Centric Sparse Interaction Fusion Network (TC-SIFN) is designed to guide the learning of resource-poor non-text modalities with the resource-rich text modality, fully leveraging the highest contribution of the text modality. A Top-K Sparse Attention Transformer is utilized as a means of modal inter action, suppressing inter-modal noise generated during modal interactions. We conducted extensive experiments on the CMU MOSI, CMU-MOSEI, and SIMS datasets, and the results show that our TC-SIFN-MCM model outperforms current methods, with a notably impressive Acc-2 score of 86.07% on the MOSEI dataset.