Real-Time American Sign Language Recognition on Edge Devices Using a Multilayer Perceptron

The ultimate purpose of human existence is often considered to be finding happiness and satisfaction. People will live happily only when they communicate their feelings, emotions and opinions others. Furthermore, the, further right to freedom of expression is a fundamental right but people who suffer from hearing problems cannot communicate. This may lead to their isolation from societal activities. To address this issue, sign language came into existence, primarily to enable communication, education, employment and social inclusion. In this paper, an efficient methodology for American Sign Language (ASL) recognition using hand gestures is proposed. To find an efficient model, several experiments were conducted on ASL datasets which consisting of images, where each image represents signs for several words, alphabets, numbers, etc., Finally, the Multilayer Perceptron model with 2 hidden layers (1024,512) outperformed other models in terms of accuracy (99.3%) in static sign classification. These studies focused mainly on the recognition of legal words, as hearing people are unable to solve their legal issues because they cannot communicate effectively with their signs to the police or lawyers to raise their issues. In this methodology, the MediaPipe framework is used to extract landmarks from high quality sign-images, and pre-processing is done with normalization techniques, ensuring consistency in scale and orientation to generalize the model for various types of users in different environments. Experimental results demonstrated high classification accuracy, validated the model's performance and presented its potential for a range of practical applications in accessibility and human-computer interaction. A separate lightweight prototype was developed to evaluate optimal deployment models for real-time offline inference with quantized and non-quantized models on mobile hardware (CPU, GPU, and TPU) as additional work in this paper.