Self-Supervised Learning for Open-Set Scene Graph Generation by Unknown-Object and Relationship Pseudo-Labels

Open-set scene graph generation (SGG) is a recent problem in the field of computer vision, which aims to detect relationships of known and unknown objects in an image, thereby enabling detailed understanding of complex scenes required by various real-world applications. Current state of the art of open-set SGG discovers unknown objects mistakenly detected as known by pretrained closed-set models. Previous studies on open-set SGG have attempted to discover unknown objects mistakenly detected as known by pretrained closed-set models. However, such a testing-time-only approach tends to overlook unknown objects dissimilar to known ones and thus fails to detect their relationships. In this paper, we propose a novel technique for open-set SGG based on a self-supervised learning approach for training an open-set SGG model. The methodology of this technique is centered around relationship-aware pseudo-labeling, which generates pseudo-labels of unknown objects and their relationships from intermediate predictions during training, thereby training the model to explicitly learn the relationships of unknown objects. We experimentally evaluated the proposed training-time technique in comparison with previous testing-time techniques in terms of standard evaluation metrics of open-set SGG. As the result, the proposed technique consistently outperformed the previous ones when combined with each SGG model, thereby demonstrating the effectiveness of our pseudo-label-based approach, which makes full use of training-time information on unknown objects and their relationships. Future directions include automation of hyperparameter tuning, performance improvement by combination with other models and training techniques, and extension to related tasks such as open-world SGG.