Gaze and Language in Action: Multimodal Interfaces for Cognitive Assistance

Read full text Summarise with VIRAT
Authors: Lorenz Krause, Michael Rice, Waqar Shahid Qureshi
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

In this paper, we present a novel software framework for portable, gaze-aware assistance, aimed at supporting people with dementia during daily kitchen tasks. Unlike traditional systems that rely on fixed infrastructure, our approach enables real-time, context-sensitive guidanceusingahostdevice(e.g.,a laptop)together with Meta’s Project Aria smart glasses. The core contribution lies in a multimodal sensor fusion architecture that combines egocentric vision (YOLOv8n-based object detection), binocular gaze tracking, and inertial spatial localisation. This fusion supports persistent object awareness without requiring environmental instrumentation. Detected objects and their spatial context are structured into a knowledge graph (KG) that maps feasible user actions. A compact language model (GPT-4o-mini) leverages this representation to generate voice-based task prompts,such as reminders to turn off the stove, instructions to make tea, or explanations about kitchen items. All assistance is delivered through non-intrusive voice feedback. Preliminary evaluations indicate the feasibility of this approach for enabling lightweight, wearable, and adaptive support in unstructured indoor environments for cognitively impaired users.

Fields of Study & Subjects

Gaze Tracking and Assistive Technology Context-Aware Activity Recognition Systems Assistive Technology in Communication and Mobility

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
20 March 2026
Language
EN