The Impact of Specific Prompt Engineering Techniques on the Readability of LLM-Generated Patient Materials in Gastroenterology and Hepatology

BACKGROUND AND AIMS: Health literacy significantly impacts patient outcomes. While the average American reads at an 8th grade reading level, healthcare materials are often written above this, potentially contributing to our nation's health literacy gap. Large language models (LLM) and prompt engineering may be able to help address this gap by consistently generating materials at the recommended 6th grade reading level. Our study aims to assess how different prompting techniques affect the readability of LLM-generated materials. METHODS: We assessed the effects of five prompt techniques (Zero-Shot, Contextualized, Constrained, Meta, Persona) on the readability of LLM-generated explanations for fifteen common gastroenterology and hepatology conditions across twelve LLMs. Output (n = 2655) readability was assessed with two readability metrics (Simple Measure of Gobbledygook (SMOG) index, Flesch-Kincaid Grade Level (FKGL)), followed by significance testing and post-hoc analysis. RESULTS: No prompt technique or model consistently produced outputs at or below a 6th grade reading level when assessed by the SMOG index, the preferred metric when assessing healthcare materials (p < 0.001). However, prompts with less constraints yield less readable outputs, while prompts with more constraints yield significantly more readable outputs (p < 0.001). CONCLUSIONS: This study demonstrates the potential of LLMs as a tool in addressing America's health literacy gap, as we show prompt engineering affects the readability of gastroenterology and hepatology-related explanations. We also found limitations to this technique. Further optimization is necessary before LLMs can consistently generate patient materials without appropriate clinician oversight, but it implies prompt engineering as a tool in addressing our nation's health literacy gap.