Gesture-Controlled AR Robotic Puzzle for Computational Thinking

Computational thinking (CT) is a central goal in K–12 education, yet learners often struggle to connect abstract programs to physical consequences. We present a gesture-controlled augmented reality (AR) robotic puzzle environment that supports multi-modal programming over a shared computational state. A mobile robot navigates a physical board spatially registered to an AR puzzle world on a tablet. We conducted a formative study with two participants (an HCI professor and a computer science student) to examine usability and interaction design considerations. Results indicate that the sequencer supported efficient planning and reflection, whereas gesture-based control introduced reliability and feedback challenges, including recognition uncertainty and unclear state-to-action confirmation. Participants found puzzle authoring engaging but reported difficulty constructing meaningful challenges, suggesting a need for scaffolding. This work contributes a mixed-reality robotic puzzle platform that enables transitions between symbolic sequencing, embodied control, and learner-authored environment design, highlighting key requirements for supporting CT learning.