Explorative Grasping Referred Objects With Weak Stiffness via Vision-Language-Tactile Fusion

A robot’s ability to grasp referred objects is vital in daily human–robot interaction scenarios. A significant challenge is addressing the referring ambiguity of physical properties that are visually unobservable. This study proposes an active explorative grasping framework that integrates vision, language, and tactile modalities for grasping objects with weak stiffness. Herein, we present a referred object retrieval network that eliminates the referring ambiguity of object stiffness by vision-language-tactile fusion and contrastive learning. For stable grasping of objects with weak stiffness and unknown weights, we also introduce a real-time slip detection method for adaptive gripper control. Furthermore, we have developed a dataset containing vision-language-tactile sequences of grasping objects with different stiffness, weights, and textures. Experiments on both self-built dataset and real-world scenarios demonstrate the high retrieval accuracy and grasping success rate of the proposed methodology.