FLIRL-Net: fuzzy logic-driven important relationship learning for scene graph generation

Abstract Scene Graph Generation (SGG) is a fundamental task in computer vision. It entails classifying entities in images and predicting visual relationships among them. Existing scene graph generation research targets the identification of all possible relationships within an image. However, this approach may suffer from inefficiencies and generate redundant information, hindering the further processing of relationship data in downstream tasks. In contrast, identifying important relationships is more aligned with the practical requiements of downstream tasks. To address this issue, we introduce the Fuzzy Logic-Driven Important Relationship Learning Network (FLIRL-Net). Initially, we employ fuzzy logic to compute an importance score for each relationship. This computation accounts for factors affecting relationship significance and user requirements. Subsequently, we design the Relationship Label Loss Importance Weighting (RLIW) module. This module utilizes these importance scores as weights to adjust relationship sample losses. Such adjustments direct the model's focus towards important relationships. Thirdly, we develop the Importance-Based Entity Pair Feature PoolFormer (IEPFP) module to enhance the model's recognition of important entity pairs. Additionally, We propose metrics Important Relationship Recall (IR@K) and Important Relationship Precision (IP@K). These metrics assess the model's effectiveness in identifying important relationships. Experimental results demonstrate that our model excels at identifying important relationships in the VG150 dataset and effectively minimizes unimportant relationship outputs.