Prompt-imposed structure: a methodological evaluation of LLM-generated survey data for theory-based modeling

A key methodological concern in using large language models (LLMs) for survey research is whether their outputs are merely ‘prompt-valid’ rather than ‘theory-valid.’ This study empirically evaluates that distinction. Using a validated structural model linking perceived aging risk, attitudes toward private medical insurance, and insurance purchase intentions, it compares one human dataset with four GPT-4o datasets generated under progressively structured prompts. GPT-4o produces plausible Likert-scale responses and, when given path-informed instructions, recovers the expected mediation structure. However, without such cues, the model fails to reproduce the theory-consistent relationships that reliably emerge in human data. These findings indicate that structural coherence in LLM-generated responses primarily reflects the relational logic imposed by the prompt rather than autonomous inference. LLMs may support pretesting and instrument development, but their role in theory testing should remain supplementary, transparent, and methodologically constrained.