Real-World Interactivity Reconstruction

Human interaction with the physical world is shaped by the ability to perceive how objects can be touched, manipulated, and experienced. These interactive possibilities arise not only from an object s physical structure but also from experiential expectations, emotional relationships, and narrative contexts. As digital representations become increasingly embedded in everyday life, reconstructing real-world interactivity has become a critical challenge. However, most existing systems focus primarily on geometry or appearance, overlooking the experiential, emotional, and structural foundations that make interaction meaningful. This dissertation investigates how real-world interactivity can be understood and digitally reconstructed. It begins by establishing a theoretical framework grounded in 3D objects’ affordances, demonstrating that interactivity emerges from the relationship between object properties and human action. Building on this foundation, the research examines how people perceive interactivity in digital reconstructions and shows that interactive behaviors are essential for preserving a sense of reality. The work further explores how emotional engagement and story-telling practices shape the meaning of interactivity, revealing that interactive gestures and their resulting object effects carry expressive significance in human memory. The dissertation then develops a general structural framework for interactivity by analyzing how components, materials, and mechanical relationships define the actions an object can support. Rather than relying on object-specific categories or predefined interaction labels, this framework provides a unified, computational representation applicable across objects of diverse forms, scales, and functions. Based on this representation, the dissertation introduces methods for the automatic generation of interactive digital twins that capture the physical and semantic foundations of object behavior. Collectively, these studies characterize interactivity as an emergent property shaped by experience, emotion, and structure, rather than a feature reducible to geometry or discrete labels. Finally, the dissertation outlines future directions that position generative models as a new paradigm for interactivity reconstruction. By modeling interaction as a dynamic, probabilistic, and context-sensitive process, generative systems enable the simulation of diverse and personalized behaviors and support world models capable of reasoning about object behavior and action possibilities.