PopSignAI: Towards Using Sign Language Recognition Games to Improve American Sign Language Learning in Novice Signers

To help novice signers learn American Sign Language, we develop PopSignAI, a proof-of-concept smartphone-based bubble-shooter game that facilitates real-time interaction through isolated sign language recognition. In a 20-person user study, we demonstrate that encouraging novice signers to practice generating sign in PopSignAI is more efficient for teaching ASL skills than a version of PopSign focused on receptive signing ability. We use over 200,000 examples of 250 signs from 47 signers to train and test a user-independent LSTM recognizer that achieves 82.9% accuracy on an independent test set. For the purposes of the game, the recognizer averages 99.6% accuracy with a 7ms inference time using a 2.5MB model. Ablation studies suggest that as few as eight signers are need for training in order for adequate recognition accuracy for PopSignAI’s gameplay. To encourage future sign language recognition games, we release the PopSignAI recognition pipeline and software. We identify hearing parents of deaf children as important potential users of sign games and conduct interviews with eight of these parents, investigating their motivation and challenges in learning sign.