Sticks and Stones... Bias and Readability Assessment in Large Language Model-Generated Patient Education for Anterior Cruciate Injury

OBJECTIVE: To (1) explore sex/gender/ethnic/socioeconomic status (SES)-related bias in large language models (LLM)-generated anterior cruciate ligament (ACL) patient education materials (PEMs) and (2) assess the accuracy, consistency, and understandability of the information provided. DESIGN: Cross sectional METHODS: Four LLMs provided PEMs on ACL injury following ten unique personas. Readability was assessed via Flesch-Kincaid Grade Level. Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) assessed understandability and actionability. The Linguistic Inquiry and Word Count dictionary assessed language differences to measure sex/gender/ethnic/SES-related bias. A qualitative analysis assessed accuracy and robustness. RESULTS: Forty LLM-generated responses were analyzed. Mean Flesch-Kincaid Grade Level ranged between 9.9 (SD 0.8; Low-SES persona) and 11.4 (SD 1.5; Boy persona). Thirty-six (90%) and 27 (67.5%) responses scored above the 70% adequate PEMAT-P threshold for understandability and actionability, respectively. The Kruskal-Wallis test and Mann-Whitney U-Test revealed no statistically significant language differences between personas, across LLMs (p>.05). Linguistic analysis revealed 37 (92.5%) persona responses had a negative tone. CONCLUSION: The LLM-generated PEMs were adequately understandable with minimal sex/gender/ethnic/SES-related bias and provided actionable information. The difficult reading level may limit generalizability. Crucial information was lacking from PEMs and the negative tone may unintentionally increase patient fear.