Speech-to-trajectory: learning intuitive verbal guidance for robot motion

Abstract Enabling intuitive, low-level robot control through natural spoken language is essential for deploying service robots in everyday environments. However, verbal commands are inherently variable and often ambiguous, making it difficult for robots to consistently interpret and execute them without extensive handcrafted rules or predefined templates. In this paper, we present the directive language model (DLM), a novel speech-to-trajectory framework that directly maps unconstrained verbal instructions to low-level robot motion trajectories. DLM is trained using behavior cloning (BC) on demonstrations in simulation, where human participants issue spoken guidance and control the robot’s motion accordingly. To improve generalization across phrasings, we apply semantic augmentation via GPT-4, generating diverse paraphrases that share the same trajectory labels. A text-conditioned diffusion policy is then employed to generate smooth and flexible trajectories that align with user intent. Unlike large language model (LLM)-based approaches that require prompt engineering and often yield unpredictable responses, DLM ensures consistent motion generation with efficient inference suitable for onboard deployment. Our results, validated in both simulation and on a quadruped robot, show that DLM robustly interprets a wide range of user commands, including paraphrased, truncated, and implicit instructions, without relying on perception or symbolic planning. These capabilities make DLM well-suited for natural, speech-based control of service robots interacting with untrained users.