How Does Prompt Anchoring Affect Large Language Model Outputs?

This study examines how different prompt anchoring strategies influence the conceptual representation of LLM-generated keywords and compares those effects with the effects of model selection. A controlled exploratory experiment evaluated four prompt conditions—No Examples, Brief Keywords, Detailed Explanations, and Author-Based Examples—across T. D. Wilson’s four information behavior dimensions using 1068 abstracts. Four LLMs (GPT-4o-mini, Claude-3-haiku, Gemini-2.0-flash-lite, and DeepSeek V3) were evaluated under all prompt conditions, yielding 17,036 valid observations. Results indicate that model identity accounts for substantially more variance in keyword generation (η2 = 0.309) than prompt condition (η2 = 0.069), although these estimates should be interpreted with caution given the repeated-measures design and assumption violations. Prompt anchoring, however, consistently reconfigured the conceptual distribution of outputs across all models, indicating that it influences conceptual representation even when model effects are larger. Author-Based Examples substantially increased representation of the typically underrepresented Information Sharing dimension, whereas Detailed Explanations produced the highest overall generation rates and the broadest dimensional coverage. These findings further indicate that different anchoring strategies involve consistent trade-offs in dimensional coverage. The study thereby identifies prompt anchoring as a source of methodological variation in LLM-assisted content analysis, indicating that anchoring strategies should be explicitly specified, justified, and reported as part of the study methodology.