Learning Robot-Aware Object Affordances for Manipulation

A key challenge in autonomous mobile robotics lies in enabling robots to interact effectively with previously unseen objects across diverse tasks. Addressing this problem requires a profound understanding of object functionality to identify actions that are both feasible and lead to the intended outcomes. The concept of affordances provides a powerful framework for this understanding. Affordances describe the potential actions that an agent can perform with an object, determined jointly by the object’s physical and functional properties and by the agent’s own morphology and control capabilities. This thesis examines different approaches to affordance learning aimed at enabling effective object interaction in robotic systems. In contrast to conventional object-centric approaches, we introduce robot-centric learning strategies that incorporate the robot’s morphology and control capabilities. This perspective enables affordance prediction and action planning that are tailored to the capabilities and constraints of a specific embodied agent, resulting in feasible, robust, and adaptive interaction strategies. In the first part of this work, we present a simulation-based framework for affordance learning, with a focus on articulated object manipulation tasks such as door opening and closing. We propose a novel closed-loop manipulation pipeline that integrates point-level affordance inference from visual input with a whole-body, sampling-based control strategy. The proposed agent-aware network incorporates the full robot model and controller to generate interaction data in simulation. During execution, affordance inference is employed in a closed-loop manner, enabling the agent to recover from failures and unexpected states. We show that this capability contributes to bridging the sim-to-real gap and improves task success rates in articulated object manipulation. In the second part of this work, we shift the focus from low-level, point-wise affordances of isolated objects to the prediction of object-level affordances within complex scenes. Understanding the potential interactions a robot can perform with objects in its environment is essential for effective high-level planning. To address this, we propose an approach for learning robot-centric affordances through interactive exploration. The method integrates an object-level mapping module into a reinforcement learning–based exploration framework, enabling the agent to track and re-identify object instances across different viewpoints. This capability enhances exploration efficiency and results in denser affordance annotations, thereby improving the overall performance of affordance prediction. In the third part of this work, we investigate an alternative to simulation-based affordance learning, which is often computationally expensive, by exploiting the capabilities of pretrained Vision-Language Models (VLMs). While these models exhibit strong contextual reasoning about objects and possible actions, they lack awareness of the local, task-specific physical setup encountered by a real robot. To bridge this gap, we propose a calibration method based on conformal risk control to estimate model uncertainty using data collected from the target environment. Real-world experiments involving the manipulation of fragile objects demonstrate that the proposed approach enables the robot to adapt its manipulation strategy according to the inferred object properties. This adaptability leads to a favorable trade-off between speed and safety, outperforming baseline methods. Finally, we investigate affordance learning in the context of tasks that require precise interactions between multiple objects, such as hanging or stacking, rather than solely between the robot and a single object. We review state-of-the-art approaches and identify key limitations, including task-specific design, overfitting to specific object categories, and extensive training requirements. This analysis underscores the need for more generalizable methods capable of jointly learning relevant geometric and physical properties to enable robust understanding of multi-object interactions. In summary, this thesis presents a comprehensive exploration of how affordances can be learned, represented, and exploited in robotic systems, with an emphasis on explicitly incorporating the robot’s embodiment into the learning process. To this end, multiple approaches are explored, including finetuning large pretrained models with real-world data and simulating the complete robot during training. The findings demonstrate that explicitly accounting for the robot’s morphology improves performance across a range of tasks, ranging from articulated object manipulation to scene-level affordance prediction and property-aware interaction. Furthermore, we analyze the requirements and review current state-of-the-art methods for tasks involving precise geometric interactions between multiple objects. The thesis concludes with a discussion of current progress in affordance learning learning and an outlook on promising directions for future research.