AI-Augmented Interfaces for Educational Storytelling

Storytelling serves as a fundamental pillar of education, acting as the primary vehicle for learning through classic mediums such as text, diagrams, or spatial maps. While modern pedagogy emphasizes the value of dynamic learning through interactive simulations, animated videos, and newer modalities like mixed reality, creating these experiences requires specialized technical expertise that most teachers lack or do not have time for. Consequently, teachers are often restricted to passive, static formats like textbooks, although this primary source of teaching is supplemented with external dynamic content. This thesis investigates how AI-augmented interfaces can democratize this process, empowering educators to transform legacy materials into interactive simulations, author original animated narratives, and enable students to experience content through emerging mixed reality modalities. This thesis follows three distinct paths toward designing and en-gineering AI-driven interfaces for educational storytelling. First, we explore a content scaffolding paradigm by addressing the legacy of static diagrams, and present Augmented Physics, a system that enables teachers to semi-automatically transform textbook diagrams into embedded, interactive physics simulations in real time. Second, we explore the narrative scaffolding paradigm by empowering the creation of original visual stories on maps, and we introduce MapStory, a generative tool that allows novices to author com-plex, controllable geospatial animations directly from free-form natural language scripts. Finally, we explore the spatial scaffolding paradigm by looking toward the future of text and situated learning, and we explore MR reading assistance by iteratively developing RealitySummary, which leverages mixed reality to de-liver on-demand, context-aware summaries anchored to the learner’s physical environment. Together, these explorations illustrate the potential of AI-augmented interfaces to evolve educational storytelling: moving from the digitization of existing materials to the generative creation of new narratives, and finally to the ubiquitous augmentation of the physical world with rich educational information on-demand.