Avatron: 3D Face Avatar and Voice Generation From Text

Speech-driven 3D facial animation models are piv otal components in a variety of human interaction systems. However, speech-driven methods are strongly dependent on the quality of input speech, and there are no ways of flexibly changing the content of the speech without re-recording, which can be costly and time-consuming. Using text-to-speech (TTS) as an intermediate step offers an avenue to generate 3D avatars for any content, but involves an inefficient pipeline: converting text to speech and subsequently generating a 3D avatar from the synthesized speech. In order to overcome this constraint, we propose a novel approach for 3D facial animation that effectively merges these two domains. Our model takes advantage of intermediate features derived from the alignment of text and speech, which are then utilized as input for an avatar decoder to transform them into the vertices of a 3D facial animation. By jointly modeling lip movements, text, and speech characteristics, we are able to achieve exceptional performance while substantially reducing the number of model parameters and computational complexity. We demonstrate the efficacy of our model by assessing it in terms of the mean square error of 3D mesh vertices and through a user study that includes ABX test comparisons.