Dataset for LLMs translation quality assessment

This study investigates and compares the performance of two state-of-the-art LLMs, namely, DeepSeek-V3 and ChatGPT-4o, in rendering into English a group of traditional Chinese legal culture-specific items (LCSIs), and examines to what extent prompt engineering can improve their quality of translation. Fifty authentic Chinese sentences containing fifty-six traditional LCSIs were selected as the source texts and translated by the two models, whose output was assessed manually based on a translation quality assessment (TQA) model featuring accuracy and understandability. A multi-stage prompt optimization process was adopted to elicit the models’best possible performance, and Wilcoxon Signed-Rank Tests were used to compare the data. The results show that there was no significant difference in the translation quality of the two models, though DeepSeek performed slightly better in the initial stage. Prompt optimization significantly improved the performance of both models, highlighting the importance of human-guided prompting. Without prompt optimization, only 25% of the LCSIs were correctly translated by both models, but with human feedback and prompt refinement, the rate of correct translation increased substantially. These findings indicate that, although current LLMs show strong potential in translating culturally embedded legal content, high-quality translation in this specialized domain still relies heavily on human expertise.