Comparative evaluation of TTS and vocoder architectures in lip-to-speech synthesis

This paper presents a comparative evaluation of model architectures in Lip-to-Speech (L2S) synthesis, a technique that reconstructs speech waveforms directly from lip motion videos. Building on the Lip2Wav framework, we examine the impact of different model components on the quality of generated speech. Specifically, we evaluate four model configurations by combining two types of Text-to-Speech (TTS) models—Tacotron2 and FastSpeech2—with two types of speech synthesis models—Griffin-Lim and HiFi-GAN. Experiments were conducted on both English and Japanese datasets, and the resulting synthesized speech was evaluated using multiple objective metrics, including STOI, ESTOI, and PESQ, to assess intelligibility and perceptual quality. Results show that the Tacotron2 + Griffin-Lim configuration achieved the highest scores in terms of intelligibility and naturalness, while the FastSpeech2 + HiFi-GAN configuration demonstrated superior acoustic fidelity and subjective quality. Furthermore, pretraining on Japanese datasets led to improved performance on English test data, indicating a positive effect of cross-lingual transfer.