From Structure to Semantics: Hypergraph-Based AR Assembly Guidance with LLM-Mediated Narration

View at publisher Summarise with VIRAT
Authors: Xinda Liu, Bowei Zhang, Jiaju Xu, Jian Wu, Guohua Geng, Lili Wang
Cited by: 0
References: 65

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Effective Augmented Reality (AR) guidance for complex assembly faces a dual challenge: the inability of conventional liaison graphs to represent procedural logic, and the cognitive burden imposed by visual instructions. We argue that the solution requires a more expressive structure to overcome these representational deficits and a narration approach to mediate instruction complexity. Our method first employs an assembly hypergraph to capture the task's hierarchical information, from which an A* search algorithm generates an optimal assembly path. Then a Large Language Model (LLM)-mediated narration workflow is designed to address the ergonomic deficiencies of the machine-centric path. It employs an optimizer to improve fluency, followed by a narrator that crafts the steps into an intuitive instruction narration. A within-subjects user study (N = 24) revealed a progressive enhancement from our method's components. The transition from a liaison-graph baseline to the hypergraph alone improved objective outcomes by reducing task time and errors and improving subjective ratings (SUS, NASA-TLX, TAM, and ARI). Subsequently, augmenting the LLM-mediated narration maintained these gains while lowering cognitive load and elevating user experience and usability. Our findings indicate the value of our AR assembly design and discuss the opportunities of using LLM as a mediation layer for better user interaction.

Fields of Study & Subjects

Augmented Reality Applications Interactive and Immersive Displays Data Visualization and Analytics

Publication details

Journal / source
IEEE Transactions on Visualization and Computer Graphics
Volume
32
Issue
5
Pages
4688–4698
Publication date
1 April 2026
Language
EN