Enhancing Fine-Grained Sketch-Based Image Retrieval Through Contextual Information

Fine-grained sketch-based image retrieval aims to retrieve matched natural photos at instance level based on hand-drawn sketches. Understanding sketches and aligning them with photos in the joint embedding space are the primary challenges. Previous works have attempted to improve the understanding and alignment of fine-grained semantics by leveraging a selected subset of local features based on the local matching strategy. However, these approaches rely heavily on local features to capture regional visual details, which introduces unavoidable uncertainty and noise in the matching process. In addition, existing works lack the ability to effectively separate unmatched pairs with similar overall semantics in the common space, which may disrupt fine-grained semantic alignment. In this paper, we presentCIGSPA: aContextual Information Guided Sketch-Photo Alignmentframework that leverages latent contextual information to enhance sketch understanding and sketch-photo matching without additional knowledge and prior supervision. Specifically, we first design anAdaptive Local Feature Filtermodule to refine feature similarity based on the context of local features and select more appropriate features, which ensure the semantic fidelity and spatial compactness of salient features. Secondly, we design aContextual Weighted Contrastiveloss to guide global sketch-photo matching by leveraging discriminative contextual information from photos, and hence aligns the sketch and photo at the instance level. Our comprehensive experiments on six public datasets demonstrate the effectiveness of our proposed CIGSPA. The code is available athttps://github.com/jinyuxu-whut/CIGSPA4FG-SBIR.