Efficient Visual Manipulation Relationship Reasoning With Relationship Attention and Sparse Graph in Robotic Grasping

Determining the reasonable grasping order and reducing the interference to surrounding objects are critical for robotic grasping under mutually-stacked scenes. However, existing manipulation relationship reasoning methods generally predict object relationships from detection perspective, and require dense evaluation of all object pairs, leading to restricted reasoning accuracy caused by detection bias and relationship class imbalance. To solve this problem, we propose an EFficient Visual Manipulation Relationship Reasoning Network, called EF-VMR2N, with relationship attention and sparse graph. Specifically, relationship attention constructs two new attention mechanisms to holistically leverage the correlation characteristics between visual features and semantic information among object pairs by highlighting reasoning-specific features. Sparse graph infers the mutual relationships within the scene using fixed number of strongly-correlated triplet sets to improve the efficiency and accuracy of object pair evaluation. Extensive experiments on the VMRD and REGRAD datasets both show that the proposed EF-VMR2N achieves SOTA performance in terms of four evaluation metrics (mAP, OR, OP and IA), and compared with object detection, the improvements on manipulation reasoning are more remarkable. The stacked objects grasping under real-world scenarios further proves the effectiveness of the proposed method. Video: https://github.com/LiMing336/VideoDemonstration.