解决机器翻译评估中的指标干扰问题，提升模型性能判断准确性。
Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation
- 提出MINTADJUST方法，缓解训练与评估使用相关指标导致的干扰
- 在WMT24测试集上，对多数语对排序更贴近人工评价，尤其对高质量系统有效
- 适用于依赖自动指标评估的翻译模型研发者，避免误判性能
随着自动评估指标日益强大且广泛应用，模型开发过程中无意间'迎合指标'的风险上升。这种现象源于指标干扰（MINT），即在模型调优和评估中使用相同或相关指标。MINT会误导从业者高估系统性能：因输出受干扰指标影响，其估计质量与人工判断的相关性下降。本文分析了机器翻译任务中两种常见MINT场景：训练数据过滤和基于质量信号的解码。研究发现，即使未直接优化指标，MINT仍严重扭曲实例级评分。为此，提出MINTADJUST方法，实现更可靠的评估。在WMT24机器翻译共享任务测试集上，MINTADJUST在多数语言对上比现有先进指标更准确地排序译文与系统，且优于主办方采用的AUTORANK集成方法。
原文摘要 · Abstract (English)
As automatic metrics become increasingly stronger and widely adopted, the risk of unintentionally "gaming the metric" during model development rises. This issue is caused by metric interference (MINT), i.e., the use of the same or related metrics for both model tuning and evaluation. MINT can misguide practitioners into being overoptimistic about the performance of their systems: as system outputs become a function of the interfering metric, their estimated quality loses correlation with human judgments. In this work, we analyze two common cases of MINT in machine translation-related tasks: filtering of training data, and decoding with quality signals. Importantly, we find that MINT strongly distorts instance-level metric scores, even when metrics are not directly optimized for-questioning the common strategy of leveraging a different, yet related metric for evaluation that is not used for tuning. To address this problem, we propose MINTADJUST, a method for more reliable evaluation under MINT. On the WMT24 MT shared task test set, MINTADJUST ranks translations and systems more accurately than state-of-the-art metrics across a majority of language pairs, especially for high-quality systems. Furthermore, MINTADJUST outperforms AUTORANK, the ensembling method used by the organizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。