EditEmoTalk: Controllable Speech-Driven 3D Facial Animation with Continuous Expression Editing

Read full text
Authors: Diqiong Jiang, K. J. Zhu, Dan Song, Jian Chang, Chenglizhao Chen, Zhenyu Wu
Cited by: 0

Abstract

Speech-driven 3D facial animation aims to generate realistic and expressive facial motions directly from audio. While recent methods achieve high-quality lip synchronization, they often rely on discrete emotion categories, limiting continuous and fine-grained emotional control. We present EditEmoTalk, a controllable speech-driven 3D facial animation framework with continuous emotion editing. The key idea is a boundary-aware semantic embedding that learns the normal directions of inter-emotion decision boundaries, enabling a continuous expression manifold for smooth emotion manipulation. Moreover, we introduce an emotional consistency loss that enforces semantic alignment between the generated motion dynamics and the target emotion embedding through a mapping network, ensuring faithful emotional expression. Extensive experiments demonstrate that EditEmoTalk achieves superior controllability, expressiveness, and generalization while maintaining accurate lip synchronization. Code and pretrained models will be released.

Fields of Study & Subjects

Human Motion and Animation Face recognition and analysis Speech and Audio Processing

Publication details

Journal / source
arXiv (Cornell University)
Publication date
15 January 2026
Document type
Preprint