SiaTalker: Siamese Emotion Injection for Coarse-to-Fine Speech-Driven 3D Facial Animation

During communication, people inadvertently express emotions through facial animations, with emotion constantly fluctuating throughout the conversation. Existing speech-driven facial animation works predominantly rely on sentence-level coarse-grained emotion labels to impart emotional expression information, thereby overlooking temporal emotion variations. How to make 3D talking heads articulate fluently while adeptly expressing mood fluctuations is the problem addressed by our work. We propose a framework, named SiaTalker, which exquisitely refines emotions of facial animations from coarse-grained levels to dual-perspective interactive fine-grained levels, resulting in highly coordinated and fluid emotional expressions. Specifically, we propose the coarse-to-fine dual granularity fusion decoding framework to connect global and local emotional aspects, allowing nuanced shifts within a consistent emotional tone. The pseudo-siamese cross-perspective submodule is embedded to interact with siamese fine-grained emotions across perspectives, further capturing subtle emotional fluctuations and amplifying the authenticity of facial movements. Furthermore, to comprehensively evaluate our work, the CREMA-D dataset is reconstructed into a 3D emotional talking face dataset with the largest number of subjects, serving as a valuable experimental supplement. Extensive experiments and user studies demonstrate that our approach outperforms state-of-the-art methods and exhibits superior emotional agility and expressiveness in facial movements.