TransZSIS: Superpixel-Guided Irregular Patch-Pair Features Learning With Transformer for Zero-Shot Instance Segmentation in Robotic Environments

Object instance segmentation is a key prerequisite for service robots to perform daily chores in unstructured environments. Traditional supervised learning-based segmentation solutions rely on massive annotated datasets, which are impractical for the wide variety of objects in real-world scenarios. To this end, we propose a novel zero-shot instance segmentation approach (TransZSIS) that enables precise instance segmentation without relying on external semantic embeddings or auxiliary information to address the unseen object instance segmentation (UOIS) problem. First, the RGB and depth images are segmented into irregular patches based on a super-pixel segmentation algorithm to generate a unified segmentation map, and then the comprehensive feature vectors of each patch is extracted and paired. Further, a Transformer-based architecture is introduced to capture the correlation between different patch-pair and the intrinsic characteristics of each patch-pair. To predict patch-pair relationships, TransZSIS uses a four-layer fully connected neural network (FCNN) to classify the transformer-encoded features and refine them with a graph-based processing tactic to achieve object instance segmentation. Extensive evaluations on both synthetic and real datasets demonstrate that TransZSIS achieves superior performance compared with state-of-the-art baseline methods. Also, we implement real experiments to verify that our solution can achieve robot grasping by segmenting unseen objects.