GestureTalk: A Real-Time CNN and LSTM Based Framework for Two-Way Indian Sign Language Communication

Read full text Summarise with VIRAT
Authors: S. Rajalakshmi, Pratyusha Kumar Pati, K. Patchaivalliammal, S. Sreepadh Krishnan
Cited by: 0
References: 5

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Communication barriers remain a significant challenge for individuals with hearing impairments, especially in everyday face-to-face situations.To address this issue, GestureTalk has been developed as a dual-mode mobile and web application that enables real-time interpretation between Indian Sign Language (ISL) and spoken or written language.The system employs a live video stream to track hand movements using MediaPipe Hands for landmark extraction, followed by a hybrid Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) model to interpret dynamic gesture sequences.The proposed model achieves an accuracy of 97.92% with an average inference time of 35 ms per gesture on mobile devices, surpassing benchmarks such as VGG16 (95.56%) and AlexNet (93.90%).Conversely, text or speech input is rendered into animated ISL gestures, enabling smooth two-way visual communication with a response time of 40 ms.By integrating gesture recognition into video calls through WebRTC, GestureTalk delivers low-latency, secure, and accessible communication.The framework demonstrates high precision (96.85%), recall (98.87%), and F1-score (95.86%), indicating its effectiveness as an inclusive assistive technology for the hearing-impaired community."

Fields of Study & Subjects

Hand Gesture Recognition Systems Hearing Impairment and Communication Human Pose and Action Recognition

Publication details

Journal / source
Advances in intelligent systems research/Advances in Intelligent Systems Research
Pages
848–871
Publication date
1 April 2026
Language
EN
Document type
Book chapter