Indistinguishable Narratives: A Blinded Comparison of AI- Versus Human-Written Personal Statements in Application to Orthopaedic Surgery Residency

ABSTRACT Background As residency program applications become more homogenized due to pass-fail United States Medical Licensing Examination Step 1 scores, pre-clinical grading, and standardized letters of recommendation, personal statements remain a key subjective tool to assess applicant fit. With the rise of large language models like ChatGPT, applicants may increasingly use AI to draft these statements. Objective To determine whether ChatGPT can generate orthopaedic surgery personal statements that are indistinguishable from and comparable in quality to human-written statements. Methods In 2025 at a single urban academic university hospital, ChatGPT-4.0 was given a standardized prompt to generate 5 personal statements across distinct themes. Five de-identified statements from applicants who interviewed or matched at our institution from 2020 to 2021 were randomly selected. The 10 essays were randomized and independently reviewed by 8 blinded assessors, including 6 attending physicians and 2 resident physicians, all of whom serve on the resident admissions committee. Using a 9-question REDCap survey, evaluators rated each statement on readability, originality, authenticity, and overall quality on a 100-point scale; guessed whether each was AI- or human-generated (Yes/No); and indicated whether they would offer an interview (Yes/No). Chi-square tests were used for analysis. Results There were no significant differences between ChatGPT and human-written statements in readability ( P =.63), originality ( P =.48), authenticity ( P =.79), or overall quality ( P =.84). Assessors could not reliably identify AI- versus human-written statements (19 of 40 vs 22 of 39, P =.43) and interview offers were similarly comparable (29 of 40 vs 29 of 39, P =.85). Conclusions ChatGPT can produce orthopaedic surgery personal statements that are both indistinguishable from and comparable in quality to human-written ones.