Multimodal Sentiment Analysis of Twitter Data Using Early Fusion Along with a Fully Connected Neural Network and Multilayer Perceptron Framework

Multimodal sentiment analysis of social media data remains challenging due to the complex integration of textual and visual information. A Fully Connected Neural Network and Multilayer Perceptron-based Sentiment Analysis (SA-FCNNMP) framework is proposed for effective multimodal sentiment classification of Twitter data. The approach combines textual and visual data to improve sentiment comprehension and is tested on the Multi-View Sentiment Analysis (MVSA) Single Twitter Dataset, which consists of 4,869 images labeled with their corresponding texts. The approach begins by cleaning textual data through stop-word removal using Bayesian Boundary Trend Filtering (BBTF), preserving only meaningful words. Subsequently, features are extracted from text and images independently to leverage their unique properties. The input textual data are represented using Word2Vec embeddings that encode semantic relationships, whereas image features are extracted using a ResNet-50 convolutional neural network. These features are then fused at an early stage using Hierarchical Multi-Scale Feature Fusion (HMSFF), which incorporates information across all scales and modalities into a consistent representation. The fused features are channeled through a Fully Connected Neural Network and Multilayer Perceptron (FCNNMP), which is optimized using the data to classify sentiment more effectively. The SA-FCNNMP model categorizes sentiments as positive, negative, or neutral. Performance is evaluated using accuracy, precision, recall, sensitivity, and computational time. To maximize learning and prediction robustness, the model uses both cross-entropy loss, which works well for multi-class classification problems, and Mean Squared Error (MSE) loss, which captures finer-grained variations in sentiment distribution. Compared with existing state-of-the-art methods, experimental results demonstrate that the proposed SA-FCNNMP model outperforms existing methods in multimodal sentiment analysis on Twitter data.