AirTalk: Real-Time Air Gesture Recognition for TalkBack Navigation

The touch-free interaction in real-time can be of much help to the mobile users who have tremors, limited mobility, or cannot use precise touch gestures. This paper describes AirTalk, a system of air-gestures navigation, which can be used to activate the main TalkBack features of Androidbased devices with the help of the front-facing camera. The system takes short gesture sequences and processes those through a lightweight CNN-LSTM model and translates those to important screen reader actions like directional navigation, scrolling, and item activation. Curative selection of gestures has been done to guarantee clarity and ease of performance particularly in patients with restricted motor control. Experimental analysis shows that there is a general recognition rate of 92.4% and the average inference time is 38.6 ms, which is appropriate to run the system in a continuous on-equipment mode. The preliminary user study on the motor limited individuals can reveal that there are an enhanced rate of task completion and a minimized physical burden when using the touch-based TalkBack controls. The findings point at the future success of camera-based air gestures as a type of input accessible to the users of screen readers.