AcouLetter: Enabling Fine-Grained Mid-Air Alphabetic Thumb Gesture Recognition on Smartphones with Acoustic and Inertial Sensing

Mobile interaction is fundamentally tied to the screen, requiring users to look at and physically touch their devices by tapping on-screen controls. We introduce AcouLetter, a system for recognizing fine-grained, mid-air alphabetic thumb gestures above the screen that enables one-handed and non-contact input. Our method fuses acoustic signals with inertial sensor data to discern the subtle characteristics of each letter gesture, capturing the dynamics of the hand’s motion. We employ a multi-stage architecture that integrates a convolutional network for acoustic features and a transformer-based network for inertial data. AcouLetter first detects the initiation of the gesture using inertial measurements, determines which hand holds the phone, and then classifies the specific letter among 27 classes, including the 26 letters of the alphabet and a null gesture. The system achieves high accuracy and demonstrates robust performance even under ambient noise while a user is standing or walking. Since letter-shaped gestures are meaningful and readily recognizable, they support an intuitive system of expressive shortcuts in which letters act as mnemonics for complex functions. A user study further confirmed the practicality of this approach for one-handed, contact-free interaction. Our results demonstrate that this expressive alphabetic gestural interface is a feasible new modality for commodity smartphones.