Preprocessing Strategies for Sentiment Analysis of Code-Mixed Social Media Text

In social media conversations, users often mix English with their native language, resulting in code-mixed (CM) text that exhibits high variability in writing style, vocabulary, and grammar. Transliteration, where native-language words are written using the Latin script, leads to multiple spelling variants of the same word, causing vocabulary fragmentation, weakened sentiment cues, and increased out-of-vocabulary issues for deep learning models. Additionally, CM sentiment datasets frequently contain emojis that convey sentiment even in otherwise neutral statements, and multiple visually distinct emojis may express the same sentiment, similar to transliteration variants. To address these challenges, this paper proposes a preprocessing pipeline incorporating transliteration normalization using Levenshtein distance, and emoji mapping using a custom token dictionary to strengthen the sentiment cues. An XLM-R classifier was implemented for sentiment analysis on Malayalam-English CM data, and the proposed preprocessing strategies were applied to improve the model robustness. Experiments on the DCM dataset demonstrate a 2.8 % relative improvement over the baseline classifier, and a 17.46 % relative improvement over the existing best results. Detailed error analysis confirms the generalization capability of the proposed approach.