用细粒度错误标注微调翻译模型，提升翻译质量。
Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
- 基于片段级错误标注设计微调算法TWA，精准定位错误区域。
- 在英德、中英翻译任务上优于SFT和DPO等基线方法。
- 适合需要高精度翻译的场景，如专业文档翻译。
尽管越来越多研究关注通过反馈改进语言模型，但多数工作仅使用序列级标注。本文探索利用离线数据集中的细粒度片段级标注来提升模型质量。我们提出一种简单微调算法——带标注训练（TWA），直接在标注数据上训练机器翻译模型。TWA利用针对性的片段级错误信息，同时灵活学习如何在片段内进行惩罚。此外，TWA在决定哪些非错误片段可作为正向信号时，会考虑序列的整体轨迹。在英语-德语和中文-英语机器翻译任务上的实验表明，TWA在经过质量筛选的序列上优于监督微调（SFT），在相同数据构建的成对样本上也优于直接偏好优化（DPO）。
原文摘要 · Abstract (English)
Despite growing interest in incorporating feedback to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of utilizing fine-grained span-level annotations from offline datasets to improve model quality. We develop a simple finetuning algorithm, called Training with Annotations (TWA), to directly train machine translation models on such annotated data. TWA utilizes targeted span-level error information while also flexibly learning what to penalize within a span. Moreover, TWA considers the overall trajectory of a sequence when deciding which non-error spans to utilize as positive signals. Experiments on English-German and Chinese-English machine translation show that TWA outperforms baselines such as Supervised FineTuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。