Towards Embodied Agents that See, Simulate, and Reason

Large language models have revolutionized textual reasoning, yet their ability to act meaningfully in multimodal, real-world environments remains limited. They struggle to ground their decisions in visual context, adapt to changing goals, and plan actions over time—shortcomings that stem from a lack of structured, goal-driven reasoning and insufficient representations of the physical world. In this talk, I present a unified framework for building embodied agents that can see, simulate, and reason. I begin by introducing methods for learning world simulators from data, arguing that visual reasoning—like textual reasoning—benefits from step-by-step processing. Inverting a physics simulator becomes key: agents must infer structured 3D neural representations of objects, parts, motions, and scenes directly from raw video. I describe methods for extracting such representations using generative priors, injecting them into vision-language models (VLMs), and scaling up their supervision via generative 3D simulation and fast, modular physics engines. These simulators enable agents to anchor their predictions in grounded physical reality, reducing hallucinations and improving control. Complementing this simulation capability, I explore techniques that enable agents to reason over time and adapt their behavior. By integrating structured memory systems, agents learn to retain and retrieve relevant experiences to inform long-horizon plans. Language-based reflective feedback allows them to refine their strategies beyond what sparse rewards offer, forming abstractions that generalize across tasks. When trained to ground their reasoning directly in the visual environment, agents gain the ability to set subgoals, explore effectively, and verify their own hypotheses. Together, these advances point toward autonomous systems that simulate before they act, reflect after they fail, and maintain an ongoing awareness of goals, constraints, and context. I will illustrate these capabilities across web automation, robotics, and interactive assistance, showing how agents that see, simulate, and reason offer a promising path toward general-purpose embodied intelligence.