用成对比较提升机器翻译质量评估准确率
PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation
- 将翻译质量评估转化为成对对比任务，预测优劣与差距
- 在WMT24上超越参数量更大的基线模型，性能更优
- 适合用于翻译解码优化，降低计算成本
我们提出PEAR（Pairwise Evaluation for Automatic Relative Scoring），一种基于监督学习的质量评估（QE）方法族，将无需参考的机器翻译（MT）评估重构为分级成对比较。给定源句和两个候选译文，PEAR预测两者质量差异的方向与大小。该方法使用人类评分差异生成的成对监督信号进行训练，并引入正则化项，确保候选顺序反转时预测符号相反。在WMT24元评估基准上，PEAR优于使用相同数据和骨干网络训练的单候选QE基线，验证了成对建模的优势。尽管参数量远小于近期大型指标，PEAR仍超过多个更大规模的QE模型及参考基准。分析表明，PEAR提供的评估信号冗余度更低。此外，我们证明PEAR可作为最小贝叶斯风险（MBR）解码的有效效用函数，在几乎无损的前提下显著降低成对评分开销。
原文摘要 · Abstract (English)
We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised quality estimation (QE) metric family that reframes reference-free machine translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for minimum Bayes risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。