MERGE: A Bimodal Audio-Lyrics Dataset for Static Music Emotion Recognition

The Music Emotion Recognition (MER) field has seen steady developments in recent years, with contributions from feature engineering, machine learning, and deep learning. The landscape has also shifted from audio-centric systems to bimodal ensembles that combine audio and lyrics. However, a lack of public, sizable and quality-controlled bimodal databases has hampered the development and improvement of bimodal audio-lyrics systems. This article proposes three new audio, lyrics, and bimodal MER research datasets, collectively referred to as MERGE (for Music Emotion Recognition next GEneration, the research project that funded its creation), which were created using a semi-automatic approach. To the best of our knowledge, this is the largest bimodal quality-controlled static MER dataset. Furthermore, the dataset is annotated both with discrete emotion categories (according to Russell's four emotion quadrants) and continuous arousal-valence values. To comprehensively assess the proposed dataset and establish a baseline for benchmarking, we conducted several experiments for each modality, using feature engineering, machine learning, and deep learning methodologies. Additionally, we propose and validate fixed train-validation-test splits. The obtained results confirm the viability of the proposed datasets, achieving the best overall result of 81.74% F1-score for bimodal classification.