Human-in-the-loop RGB semantic segmentation for intuitive teaching of novel grasping tasks via few-shot adaptation of lightweight neural networks

Assistive robots are starting to be a part of everyday human life, yet the cognitive abilities of the robots still need to be improved to successfully operate in human environments. Vision-based grasping systems require determining the correct semantic region in the image that is suitable for task-oriented grasping of target objects. The complexity of unstructured environments like households and healthcare facilities necessitates the ability to learn new segmentation tasks from human guidance and to dynamically extend the perceptual skills on a daily basis to improve autonomy. We propose to use a browser-based deep interactive RGB segmentation interface for receiving the human guidance intuitively, allowing non-experts to guide the robots from remote. We further leverage meta learning to quickly adapt lightweight neural networks for custom semantic segmentation tasks with a few human demonstrations, which facilitates a quick, cost-efficient onboard learning on the assistive robots. For training and evaluation of the few-shot learning module with grasp area affordances, we extended a largescale generic grasping dataset with affordance segmentation labels and created a new dataset with 82,944 cluttered scene images and the corresponding segmentation masks. Comparative experiments on the datasets indicate the effectiveness of our few-shot learning modules by reaching high segmentation accuracies at a low computational cost. In addition, the robotic experiments with a 7-DOF manipulator show that the proposed methods outperform the baseline method by a large margin (22 to 25%) in task-oriented grasp precision and success rates. Lastly, the conducted user studies demonstrate the ease of use of the proposed demonstration interface. Consequently, our end-to-end pipeline facilitates real-world deployment of assistive robots in human environments by intuitively teaching new semantic segmentation tasks on a daily basis. TThe generated affordance dataset can be found at https://mediatum.ub.tum.de/1839309 .