The hall of mirrors: large language models as individuals or populations?

An unsolved methodological question in psychometric assessments of large language models (LLMs) is whether to conceptualise an LLM as an individual or a population from which responses can be sampled. This distinction impacts statistical analysis and interpretation, but there is no consensus on the appropriate conceptualisation yet. In this study, we will investigate whether the variance between (versus within) LLM responses to multiple questionnaires is more characteristic of repeatedly measured individuals or a population sample. We will use intraclass correlations (ICCs) derived from random effects models, treating questionnaire items as repeated measurements of a given (sub)construct. The ICC quantifies the proportion of the total variance in responses that is attributable to systematic differences across responses (i.e., between-response variance) as opposed to unsystematic variability (i.e., within-response variance or residual error), essentially indexing the extent to which the observed variance stems from individual differences versus idiosyncratic or random noise. ICCs of LLM responses will be compared to a human sample of responses to the same questionnaires, clarifying whether LLMs exhibit individual-like consistency or population-like diversity.