Two audiences, two standards: evaluating human vs. LLM poetry translation through bilingual and monolingual lenses

Abstract This study investigates how bilingual (Chinese-English) and monolingual (English) audiences evaluate classical Chinese poetry translations, comparing human and GPT-4 outputs. Using mixed methods (ratings, qualitative comments), 19 bilingual and 19 monolingual readers assessed five translations (four human, one GPT-4) for each of six classical Chinese poems. Findings revealed two distinct, audience-specific standards. Bilingual readers, applying a fidelity-forward standard, valued comprehensive faithfulness (a synthesis of semantic fidelity, poetic structure, and cultural nuance) and strongly preferred a human translation. Conversely, monolingual readers, applying a fluency-forward standard, prioritized linguistic fluency, clarity, and emotional resonance, and significantly preferred the GPT-4 translations. Qualitative analysis further illuminated these divergent criteria. These findings indicate that evaluation standards are significantly shaped by source language access. Notably, AI-generated translations can achieve a positive reception from target-language readers by excelling in prioritized dimensions like fluency and clarity, in some cases rated more favourably than human translations by monolingual readers. This research underscores the importance of audience-centric evaluation, highlighting that assessing both human and AI translations requires consideration of distinct criteria relevant to specific reader profiles. The study offers insights for tailoring translation strategies to more effectively bridge cultural gaps while preserving literary value for diverse audiences.