Affordance-Guided Robotic Grasping via Multimodal Large Language Model Reasoning

Robotic grasping is a core capability for embodied AI agents, but traditional methods focus solely on stable object holding and fail to account for object functional semantics, which limits their effectiveness in unstructured, human-centric scenarios. This gap highlights the need to integrate “affordance" (object properties indicating usable interactions) into grasping systems. To address this, we proposes an MLLM-driven affordance-guided grasping framework. Unlike existing approaches that rely on large annotated datasets or additional memory modules, the framework leverages the inherent world knowledge and commonsense reasoning of multimodal large language models (MLLMs) to infer object affordances. It comprises three key modules: 1) an MLLM-based Affordance Reasoning Module that analyzes visual-language inputs to identify graspable and avoidable regions, generating structured outputs with confidence/risk scores and normalized coordinates; 2) a Segmentation with Affordance Guidance Module that converts MLLM-inferred affordance cues into binary masks, optimized via metrics for coverage, avoidance penalty, and core-point consistency; 3) a Grasp Strategy Generation Module that refines initial grasp candidates using affordance masks to output optimal grasp poses. Experiments are conducted on the real-world robotic platform. Results show the framework effectively identifies functional regions, validating its ability to enable task-driven, safe grasping. This work advances robotic adaptability in real-world contexts and provides a foundation for extending affordance-based manipulation to more complex scenarios.