GuideMe: A VLM-Based System Assisting Independent Smartphone Learning for Older Adults

Due to age-related cognitive and physical decline, older adults face numerous difficulties when learning new functions of smartphone applications. However, older adults often struggle to ask questions clearly and follow instructions independently. Through a formative study (N=16), we identified the behaviors and challenges of older adults seeking help independently and analyzed the effective mechanism of in-person instruction. Based on these findings, we proposed GuideMe, an in-situ conversational instruction system for older adults’ application learning. GuideMe utilizes Vision-Language-Models to analyze multimodal context in users’ situations, then assists users in confirming their intentions by asking clarifying questions, and finally provides step-by-step instructions using in-situ highlight and deictic gestures. We conducted a user study (N=18) that demonstrated that GuideMe significantly reduced users’ cognitive load during learning, helped them ask questions and follow instructions efficiently, and achieved performance comparable to that of in-person instruction.