Music to dance as language translation using sequence models

Abstract Synthesising appropriate choreographies from music remains an open problem, due to the need to align musical semantics with subjective human motions. We introduce MDLT, a novel approach that frames the choreography generation problem as a translation task. Our method leverages the AIST++ and PhantomDance data sets to learn to translate sequences of audio into corresponding dance poses. We present two variants of MDLT: MDLT-T based on the Transformer architecture, and MDLT-M based on the Mamba architecture, for strong long-horizon sequence modeling. Trained on these data sets, our method enables a robotic arm to dance, and can generalize to humanoid robots through its architecture-agnostic design. Evaluation metrics, including Average Joint Error and Fréchet Inception Distance, consistently demonstrate that, when given a piece of music, MDLT excels at producing realistic and high-quality choreography. The code will be made available upon acceptance.