Evaluation of Linguistic Variability of Large Language Model Text Personalization Using Natural Language Processing

Natural Language Processing (NLP) is proposed in this paper as an evaluation framework to track linguistic patterns of Large Language Models (LLMs) outputs over time. NLP metrics were used to quantify and compare the differences in repeated generation using the same prompt. Experiment 1 examined internal reliability across 10 repeated generations of four LLMs (i.e., Claude, Llama, Gemini, ChatGPT) on 10 scientific texts tailored for a reader profile. Linear mixed-effects models showed no effect of repeated trial on linguistic features (e.g., cohesion, syntactic complexity, lexical diversity, lexical sophistication, noun-to-verb ratios, sentence length, writing style). The results indicated high short-term stability but robust linguistic patterns between LLMs, suggesting unique writing style. Experiment 2 assessed linguistic variation using standardized prompts on GPT-4o (deployed in Oct 2024 versus Jun 2025) and GPT-4.1 (deployed in Jun 2025), with adaptations tailored for four hypothetical reader profiles. Significant differences in linguistic behavior was detected across model updates over time. GPT-4o (Jun 2025) produced shorter, more cohesive outputs with reduced language variety (greater consistency), whereas GPT-4.1 (Jun 2025) produced more academic, lexically sophisticated text with higher noun-to-verb ratios and longer sentences (more formal writing style). LLMs are updated quickly and frequently, yet there is currently no standardized method for tracking their linguistic patterns. The current study demonstrated the capability of NLP-based evaluation methods for detecting meaningful linguistic changes across LLM updates.