MelodyGLM: Multi-Task Pre-Training for Structured Symbolic Melody Generation

Pre-trained language models in natural language processing (NLP) have substantially advanced music understanding and generation. However, traditional pre-training methods for symbolic melody generation are limited in their ability to capture multi-scale, multi-dimensional musical structures, primarily due to fundamental differences between textual and musical domains. In addition, the scarcity of large-scale symbolic melody datasets constrains further progress. In this paper, we introduce MelodyGLM, a novel multi-task pre-training framework tailored for structured symbolic melody generation. MelodyGLM enhances autoregressive blank infilling pre-training through melodic$n$-gram and long-span sampling strategies, which target local and global structural modeling, respectively. Specifically, we define three types of melodic$n$-grams (pitch-based, rhythm-based, and pitch–rhythm hybrid) to model multi-dimensional melodic structures and their intra- and inter-dimensional dependencies. Furthermore, we construct MelodyNet, one of the largest symbolic melody datasets for pre-training, which comprises 440,102 melody pieces extracted from nearly 1.6 million MIDI files. Both subjective and objective evaluations demonstrate that MelodyGLM outperforms previous masked pre-training methods and other symbolic music generation models in modeling melodic structure. In particular, subjective evaluations show that MelodyGLM achieves average gains of 0.75 and 0.69 points in melody continuation and inpainting tasks, respectively, across consistency, rhythmicity, structure, and overall quality.