A multimodal deep learning framework for enhancing Russian sentiment analysis

Multimodal sentiment analysis uses computer vision, natural language processing and deep learning techniques to improve the accuracy of sentiment recognition. This study develops a novel framework for the Russian language that integrates textual, speech, and visual data and aims to address the unique differences in Russian language sentiment. A transformer-based model collects contextual semantics, while a Convolutional Neural Network (CNN) processes speech data to analyze acoustic features based on spectrograms. The ResNet architecture is used to encode facial expression data. The weighted fusion mechanism dynamically integrates modality-specific representations to ensure robust and comprehensive sentiment classification. Experiments conducted on a Russian multimodal sentiment analysis dataset show that the proposed framework has a weighted accuracy of 85.4%, which is more effective than unimodal and baseline multimodal approaches. These findings suggest that the framework can exploit complementary information across modalities to resolve ambiguities in neutral or mixed emotions. In addition, the system can effectively recognize Russian emotional patterns, such as pitch changes and facial expressions, which are crucial for accurate emotion recognition. This study emphasizes the potential of multimodal approaches to facilitate sentiment analysis in linguistically and culturally diverse contexts.