UCSM-TG: Utterance, Conversation and Speaker-Level Speech Emotion Tracking Model in Conversations Using Transformer-GRU

Emotion Recognition in Conversation (ERC) based on speech signals faces significant challenges due to the contextual dependencies and dynamic interactions between speakers. Although existing Transformer models are capable of capturing long-range dependencies, they still encounter limitations in modeling the global dialogue structure and the emotional representations of individual speakers. We propose a novel model named UCSM-TG, which synergistically integrates Transformer and Gated Recurrent Unit (GRU) to track emotions at utterance, conversation, and speaker levels. The model consists of three core modules, the Utterance-level Emotion Tracking Module (UETM), the Conversation-level Emotion Tracking Module (CETM), and the Speaker Emotion Positioning Module (SEPM). These modules are designed to extract local emotional features, encode global contextual information, and track the emotional states of individual speakers, respectively. Furthermore, We improve Wav2vec 2.0 with the Convolutional Neural Network (CNN) to enhance the feature representation and employ the Focal Loss function to mitigate class imbalance in the datasets. Experimental results demonstrate that UCSM-TG achieves competitive performance, attaining weighted F1-scores of 45.21%, 57.63%, and 68.69%, respectively, on IEMOCAP, MELD, and EEIDB, respectively. Compared with the strong transformer baseline, UCSM-TG achieves weighted F1-score improvements of 3.75%, 1.43% and 1.76%, respectively, on the above datasets, outperforming most state-of-the-art ERC methods.