Reflexive UI Awareness as Host-Side Orchestration: Continuous Grounding, Interaction Normalization, and Embodiment Policy for Agentic Interface
Abstract
This paper describes the Hybrid Interaction Engine (HIE), a host-side interaction approach for embodied assistants that combine dialogue with rendered UI. The approach applies three design principles: persistent model-visible grounding of rendered state, normalization of UI interactions into conversational events, and avatar-expression arbitration. Demonstrated in Grimoire, an SPFx-based Microsoft 365 assistant with voice and text interaction, 35 runtime tools, 1 block types, and MCP server integration. The contribution is a pattern description and reference implementation, not claim of protocol-level novelty.
Fields of Study & Subjects
Speech and dialogue systems
Usability and User Interface Design
Social Robot Interaction and HRI
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 15 March 2026
- Language
- EN
- Document type
- Preprint