用质量评估模型做可反向传播的损失，提升机器翻译效果。
QE-EBM: Using Quality Estimators as Energy Loss for Machine Translation
- 将质量评估模型作为可训练损失网络，支持梯度回传。
- 在多语言翻译中均超越基线，低资源语言提升显著。
- 特别适合无标注数据丰富的低资源翻译任务。
强化学习在对齐语言模型与人类偏好方面展现出巨大潜力，尤其适用于机器翻译等文本生成任务。对于翻译任务，质量评估（QE）模型可为未标注数据生成奖励信号。然而，传统强化学习无法利用相对于QE得分的梯度。本文提出QE-EBM，将质量评估器作为可训练的损失网络，能够直接反向传播梯度至神经机器翻译（NMT）模型。我们在多种低资源和高资源目标语言上进行了实验，结果表明，该方法在所有目标语言上均优于REINFORCE、近端策略优化（PPO）及监督微调等强基线模型，尤其在低资源语言上表现突出。以英译蒙语为例，相比监督基线，分别提升了2.5 BLEU、7.1 COMET-KIWI、5.3 COMET和6.4 XCOMET。
原文摘要 · Abstract (English)
Reinforcement learning has shown great promise in aligning language models with human preferences in a variety of text generation tasks, including machine translation. For translation tasks, rewards can easily be obtained from quality estimation (QE) models which can generate rewards for unlabeled data. Despite its usefulness, reinforcement learning cannot exploit the gradients with respect to the QE score. We propose QE-EBM, a method of employing quality estimators as trainable loss networks that can directly backpropagate to the NMT model. We examine our method on several low and high resource target languages with English as the source language. QE-EBM outperforms strong baselines such as REINFORCE and proximal policy optimization (PPO) as well as supervised fine-tuning for all target languages, especially low-resource target languages. Most notably, for English-to-Mongolian translation, our method achieves improvements of 2.5 BLEU, 7.1 COMET-KIWI, 5.3 COMET, and 6.4 XCOMET relative to the supervised baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。