Open-vocabulary models for object detection and segmentation in visual art: survey and comparative study

Abstract Objects present in paintings help art history specialists interpret and decode artworks. The analysis of large, digitized artistic collections became feasible thanks to modern object detection approaches. Nevertheless, the use of object detection models typically requires fine-tuning for specific tasks. Therefore, art history specialists remain constrained by the categories of objects in existing labeled artistic datasets when using artificial intelligence methods. This limitation can be overcome by using recent models that combine two modalities: vision and text. Vision-language models have made open-vocabulary detection (OVD) possible, allowing detection without restrictions on the applied categories, in contrast to fixed-vocabulary detection. Recent literature lacks a comprehensive review focusing on OVD in artistic images. In this paper we analyze state-of-the-art models for OVD, analyze their transferability to cultural heritage categories and systematically evaluate them on artistic datasets commonly used in literature. The DEArt and IconArt datasets, which are annotated with cultural heritage-specific categories contain paintings from the 11th to the 20th century. While the Watercolor2K dataset, annotated with common object categories consists of watercolor paintings. Based on our analysis, the OWLv2 model achieved the best performance in both object detection and grounding task scenarios on these datasets. Additionally, we discuss existing challenges of open-vocabulary segmentation in artistic images and future tasks.