不依赖人工标注，用模型内部信号实现机器翻译错误的精准定位。
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement
- 利用语言模型内部可解释性与不确定性量化，无监督识别翻译错误。
- 在12个翻译方向上验证，多标注集下评估显示无监督方法性能更稳定。
- 揭示单标注者评价的脆弱性，适合追求高效、鲁棒质量评估的研究者。
词级质量估计（WQE）旨在自动识别机器翻译输出中的细粒度错误片段，广泛应用于译后编辑辅助。现有方法常需昂贵的大规模提示或人工标注训练数据。本文探索利用语言模型可解释性与不确定性量化的新路径，从翻译模型内部机制中识别错误。在涵盖14种评估指标、12个翻译方向的实验中，通过多组人工标注数据量化了标注差异对指标表现的影响。结果表明，无监督方法潜力巨大，而有监督方法在标注不确定时表现不佳，同时暴露了单一标注者评估的局限性。
原文摘要 · Abstract (English)
Word-level quality estimation (WQE) aims to automatically identify fine-grained error spans in machine-translated outputs and has found many uses, including assisting translators during post-editing. Modern WQE techniques are often expensive, involving prompting of large language models or ad-hoc training on large amounts of human-labeled data. In this work, we investigate efficient alternatives exploiting recent advances in language model interpretability and uncertainty quantification to identify translation errors from the inner workings of translation models. In our evaluation spanning 14 metrics across 12 translation directions, we quantify the impact of human label variation on metric performance by using multiple sets of human labels. Our results highlight the untapped potential of unsupervised metrics, the shortcomings of supervised methods when faced with label uncertainty, and the brittleness of single-annotator evaluation practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。