Unsupervised Memorability Modeling from Tip-of-the-Tongue Retrieval Queries

Visual content memorability has intrigued the scientific community for decades, with applications spanning from understanding nuanced aspects of human memory to enhancing content design. A significant challenge in progressing the field lies in the high cost of collecting memorability annotations from humans, which constrains both the diversity and scalability of available datasets. Existing datasets typically provide only aggregate memorability scores for visual content, overlooking the nuanced signals embedded in natural, open-ended recall descriptions. In this work, we introduce the first large-scale, unsupervised dataset designed explicitly for modeling visual memorability signals, containing over 82,000 videos paired with descriptive recall data. We leverage tip-of-the-tongue (ToT) retrieval queries from online platforms such as Reddit. We demonstrate that this unsupervised dataset provides rich signals for two memorability-related tasks: recall generation and ToT retrieval. Large vision-language models fine-tuned on our dataset outperform state-of-the-art models such as GPT-4o in generating open-ended memorability descriptions for visual content. In addition, we employ a contrastive training strategy to create the first model capable of multimodal ToT retrieval. Our dataset and models present a new research direction and provide scalable tools for advancing work on visual content memorability.1