用翻译后编辑生成偏好数据，提升大模型翻译质量
Post-edits Are Preferences Too
- 将译后编辑隐含偏好用于大模型微调
- 预训练加微调使模型更接近人工编辑风格
- 适合需要高质量翻译的场景
当前主流的大语言模型微调方法偏好优化（PO）依赖人工对成对翻译结果的偏好判断。但在机器翻译领域，此类判断难以获取且可靠性较低。本文研究译后编辑能否作为可靠偏好来源：编辑在修改时已知修改后的版本应优于原版，这种隐含偏好可直接用于训练。实验表明，利用该机制可引导模型输出更接近人工编辑风格的译文，而非原始机器翻译风格。进一步发现，先通过译后编辑进行监督微调（SFT）预训练，再进行偏好优化，能获得最佳效果。
原文摘要 · Abstract (English)
Preference Optimization (PO) techniques are currently one of the state of the art techniques for fine-tuning large language models (LLMs) on pairwise preference feedback from human annotators. However, in machine translation, this sort of feedback can be difficult to solicit. Additionally, Kreutzer et al. (2018) have shown that, for machine translation, pairwise preferences are less reliable than other forms of human feedback, such as 5-point ratings. We examine post-edits to see if they can be a source of reliable human preferences by construction. In PO, a human annotator is shown sequences $s_1$ and $s_2$ and asked for a preference judgment, %$s_1 > s_2$; while for post-editing, editors create $s_1$ and know that it should be better than $s_2$. We attempt to use these implicit preferences for PO and show that it helps the model move towards post-edit-like hypotheses and away from machine translation-like hypotheses. Furthermore, we show that best results are obtained by pre-training the model with supervised fine-tuning (SFT) on post-edits in order to promote post-edit-like hypotheses to the top output ranks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。