Rolling Memory: A New Approach to Annotation with Generative LLMs in Social and Political Research

Abstract Generative Large Language Models (LLMs) have shown promising results in text annotation tasks, which is of interest to social scientists. The most commonly used approaches, zero-shot and few-shot learning, do not sufficiently exploit the in-context learning capabilities of these models. Extant work demonstrates that allowing these models to retain memory throughout the annotation task increases performance considerably. In this article, we propose a refinement to the memory approach from Timoneda et al. (Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks. arXiv preprint: 2503.04874, 2025) that leads to significant performance gains over the original version. We implement a two-run method where we keep 100 highly informative examples from the first run in the memory at the start of a second run. During the first run, the model improves its performance as it progresses through the annotation task, a phenomenon we attribute to in-context learning. Then, during the second run, performance is high and consistent throughout the task, improving by a further 3.56% over the .standard memory approach and a remarkable 13.26% over few-shot learning with chain-of-thought reasoning, the current gold standard. These results have important implications for applied researchers looking to improve measurement accuracy in annotation tasks