解决超长文档翻译评估难题,自动对齐并多粒度打分
Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation
- 先对齐源目标句,再滑动窗口计算多粒度评分
- 在WMT上与专家评分相关性达0.929,接近人工判断
- 可生成偏好数据用于强化学习训练,提升翻译质量
大型语言模型开启了文档级机器翻译(doc-mt)的新时代,但其整篇输出挑战了传统基于句子对齐的评估方法。我们提出 extit{Align-then-Slide},一个完整的超长文档翻译评估框架。在对齐阶段,自动推断句级源目标对应关系,并重构目标文本以匹配源句数量,解决遗漏及多对一/一对多映射问题。在n-Chunk滑动评估阶段,分别计算1、2、3、4块的平均指标得分,实现多粒度评估。在WMT基准测试中,该方法与专家MQM评分的相关系数达0.929。在新构建的真实世界测试集上,结果也与人工判断高度一致。此外,该框架生成的偏好数据可用于有效进行CPO训练,或直接作为GRPO的奖励模型,均优于基线SFT模型。结果验证了本框架在准确性、鲁棒性和实用性方面的优势。
原文摘要 · Abstract (English)
Large language models (LLMs) have ushered in a new era for document-level machine translation (\textit{doc}-mt), yet their whole-document outputs challenge existing evaluation methods that assume sentence-by-sentence alignment. We introduce \textit{\textbf{Align-then-Slide}}, a complete evaluation framework for ultra-long doc-mt. In the Align stage, we automatically infer sentence-level source-target correspondences and rebuild the target to match the source sentence number, resolving omissions and many-to-one/one-to-many mappings. In the n-Chunk Sliding Evaluate stage, we calculate averaged metric scores under 1-, 2-, 3- and 4-chunk for multi-granularity assessment. Experiments on the WMT benchmark show a Pearson correlation of 0.929 between our method with expert MQM rankings. On a newly curated real-world test set, our method again aligns closely with human judgments. Furthermore, preference data produced by Align-then-Slide enables effective CPO training and its direct use as a reward model for GRPO, both yielding translations preferred over a vanilla SFT baseline. The results validate our framework as an accurate, robust, and actionable evaluation tool for doc-mt systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。