Lip sync to speech conversion using deep learning

Speech recognition based on lip movement, also known as lip-reading AI, has received significant attention due to its potential applications in silent communication. Speech recognition from using a lip sync video has several challenges, such as noisy environments or situations where the audio is unavailable. Our main goal is to convert an audioless video into text and speech using lipsync. The way we approached the project is pre-processing video frames to extract main facial features, especially focusing on dynamics near the lip region.We have utilized convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory (LSTM). This system leverages Visual Speech Recognition (VSR) and speech synthesis models to translate silent lip movements into spoken words. The lip-reading based on the project architecture was trained to identify speech from sequences of video observations of lip motions. Exact numerical results, such as accuracy, WER, and CER, are obtained; the evaluation code can be included to calculate these values. Early results suggest that the model can generate coherent word sequences, demonstrating encouraging performance for visual speech recognition. When correctly tested, its accuracy and error rates can be measured quantitatively.