Speech-Driven Immersive Analytics: GenAI-grounded Sensemaking Perspective

Research on natural interactions between humans and immersive analytics systems has increasingly focused on the role of natural language interactions (NLIs), especially speech, as generative AI (GenAI) has become popular. Speech is a natural modality that enables users to express intent with a low learning curve and allows analytic systems to infer that intent with richer contextual information. However, it remains underexplored across various fields. Existing work still has gaps in understanding natural interactions and focuses more on multimodal interactions with speech as an auxiliary modality or textual modalities. This dissertation explores this challenge through a systematic approach: (1) the examination of the interplay between users’ speech patterns, embodiment tendencies within speech, as well as observed embodied cues in conveying analytic intent; and (2) the design of methods for speech-oriented interactions between humans and intelligent analytic systems in immersive environments, specifically from a data sensemaking perspective.