Bridging the Semantic Gap: AI-Mediated Embodied Interaction for Visually Impaired Children’s Tactile Graphics Exploration

Visually impaired children (VIC) often rely on tactile graphics and Braille descriptions to access image-based knowledge, but static materials provide limited support for local semantic explanation during exploration. This article presents TactileGuide AI, a low-cost, on-device multimodal interaction system that augments physical tactile graphics with location-specific audio feedback. Using a smartphone camera, the system detects graphic elements and tracks a child’s pointing gesture; when the fingertip enters a recognized region, it plays the corresponding spoken description while preserving tactile exploration as the primary reading mode. We evaluated the system with 40 visually impaired children, five special education teachers, and technical robustness testing. In a 30-minute recognition task, children using TactileGuide AI showed higher short-term learning gains than those using equivalent Braille descriptions. The findings provide preliminary evidence for camera-based audio augmentation in child-centered accessible learning.