AI-Driven Lip-Reading System

AI Based Visualization System for Displaying Text- to-Speech and Sign Language to Individuals with Hearing Impairment is an AI/Machine Learning and Computer Vision based system that recognises silent moving lips and translates them into readable text so that people with hearing impairment can communicate with people verbally, without needing to rely on auditory signals. The system uses video from a webcam or cellphone camera to capture images and analyse those images by first identifying sub-regions of interest around the mouth and then applying a predefined series of processes to bring all frames into a common standard form with regard to lighting, frame size, and frame frequency (e.g. frame rate, etc.). Second, it uses a Hybrid Architecture made up of Convolutional Neural Networks (CNNs) to extract spatial features of the videos as part of the input to a Sequential Model (Long Short-Term Memory network [LSTM] and/or Transformer) to model the temporal evolution of the visual representations of the speaking person’s lip movements and then generate temporal-level transcriptions of those movements (in character and/or word format). The training of this model has been based on benchmark datasets (GRID and LRW) designed to capture multiple people’s speaking patterns under different conditions. Evaluation metrics include—Word Accuracy, Character Accuracy, and Word Error Rate—allowing for quantifying performance of the lip reading model. To provide real time access for lip reading support, a web-based user interface will be established, allowing streaming of live video along with displaying recognised text and confidence scores, plus optional visualisation of the attention of the model over key frames to improve overall interpretability and user confidence in the output