Reflexive UI Awareness as Host-Side Orchestration: Continuous Grounding, Interaction Normalization, and Embodiment Policy for Agentic Interface

Read full text
Authors: Nello D'Andrea
Cited by: 0

Abstract

This paper describes the Hybrid Interaction Engine (HIE), a host-side interaction approach for embodied assistants that combine dialogue with rendered UI. The approach applies three design principles: persistent model-visible grounding of rendered state, normalization of UI interactions into conversational events, and avatar-expression arbitration. Demonstrated in Grimoire, an SPFx-based Microsoft 365 assistant with voice and text interaction, 35 runtime tools, 1 block types, and MCP server integration. The contribution is a pattern description and reference implementation, not claim of protocol-level novelty.

Fields of Study & Subjects

Speech and dialogue systems Usability and User Interface Design Social Robot Interaction and HRI

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
15 March 2026
Language
EN
Document type
Preprint