Lend a Hand: Semi Training-Free Cued Speech Recognition via MLLM-Driven Hand Modeling for Barrier-Free Communication

Cued Speech (CS) is an innovative visual communication system that integrates lip reading with hand coding, designed to enhance effective communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) refers to the AI-driven process of automatically recognizing hand gestures and lip movements in CS, converting them into text. Previous work was hindered by the scarcity of data, which prevented the full training of the hand-lip modeling and fusion modules, resulting in unsatisfactory performance that significantly limited ACSR’s effectiveness. To address this issue, we have innovatively explored the capabilities of Multimodal Large Language Models (MLLMs) in recognizing hand codings in CS and propose a new Semi Training-Free paradigm for ACSR, named STF-ACSR. More precisely, we first design the CS Prompt Module (CSPM), which enables zero-shot recognition of hand movements through a training-free keyframe filtering mechanism and customized prompt engineering with MLLMs. Second, we introduce the Minimalist Fusion Module (MFM) to seamlessly integrate the recognition results into the lip reading model, effectively achieving superior recognition results. Extensive experiments have demonstrated that STF-ACSR significantly outperforms previous methods on both normal-hearing and hearing-impaired data. Implementations and more experimental results are available at: https://github.com/DennisHgj/STF_ACSR.