用问答评估翻译信息保留，比传统方法更懂长段落语义。
Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering
- 通过生成问题测试翻译能否答对关键信息，评估段落级译文质量。
- 在文学类长文本上，表现优于或媲美顶尖神经与大模型评价指标。
- 生成的问题可定位翻译错误，提升评估可解释性，适合语言研究者使用。
尽管机器翻译评估持续进步，现有自动指标仍难以捕捉跨句边界的意义保留情况。我们认为，仅依赖模仿人类判断的单一内在评分可能不足以评估长而复杂的段落翻译，需要更‘实用’的方法来考察翻译在上下文中准确传递关键信息的能力。为此，我们提出 TREQA（基于问答的翻译评估），通过阅读理解问题评估候选翻译是否准确回答源文本或参考文本中的关键信息。在需长程理解的挑战性领域（如文学文本）中，TREQA 在排名多个段落级翻译方案方面表现优异，甚至超越部分最先进的神经和大模型指标，且从未显式优化以匹配人类判断。此外，生成的问题与答案具备可解释性：实证分析表明它们能有效定位专家识别出的翻译错误。代码已开源：https://github.com/deep-spin/treqa。
原文摘要 · Abstract (English)
Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance on a single intrinsic quality score, trained to mimic human judgments, might be insufficient for evaluating translations of long, complex passages, and a more ``pragmatic'' approach that assesses how accurately key information is conveyed by a translation in context is needed. We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality by assessing how accurately candidate translations answer reading comprehension questions that target key information in the original source or reference texts. In challenging domains that require long-range understanding, such as literary texts, we show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations, despite never being explicitly optimized to correlate with human judgments. Furthermore, the generated questions and answers offer interpretability: empirical analysis shows that they effectively target translation errors identified by experts in evaluated datasets. Our code is available at https://github.com/deep-spin/treqa
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。