用后编辑任务稳定强化学习,提升大模型机器翻译效果
PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning
- 两阶段强化学习框架,用后编辑辅助优化翻译
- 在英译土语任务上达先进水平,接近DeepSeek-V3.2
- 适合追求高精度翻译的开发者与研究者
强化学习(RL)在基于大语言模型的机器翻译中展现出巨大潜力,如GRPO方法已取得显著提升;然而,翻译导向的强化学习仍面临蒙特卡洛回报估计带来的噪声信号问题,以及庞大的轨迹空间导致全局探索过度、局部优化不足。我们提出PEGRL,一种两阶段强化学习框架,利用后编辑作为辅助任务以稳定训练并引导整体优化。每轮迭代中,通过采样翻译输出生成后编辑输入,使后编辑阶段的回报估计能基于当前翻译行为进行条件化,同时兼顾全局探索与精细局部优化。特定任务加权机制进一步平衡翻译与后编辑目标贡献,形成有偏但更高效的样本估计器。在英→芬、英→土、英↔中任务上的实验显示,性能持续优于主流强化学习基线;尤其在英→土任务上,COMET-KIWI得分接近先进大模型系统DeepSeek-V3.2。代码与预训练模型已开源于GitHub与HuggingFace。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has shown strong promise for LLM-based machine translation, with recent methods such as GRPO demonstrating notable gains; nevertheless, translation-oriented RL remains challenged by noisy learning signals arising from Monte Carlo return estimation, as well as a large trajectory space that favors global exploration over fine-grained local optimization. We introduce \textbf{PEGRL}, a \textit{two-stage} RL framework that uses post-editing as an auxiliary task to stabilize training and guide overall optimization. At each iteration, translation outputs are sampled to construct post-editing inputs, allowing return estimation in the post-editing stage to benefit from conditioning on the current translation behavior, while jointly supporting both global exploration and fine-grained local optimization. A task-specific weighting scheme further balances the contributions of translation and post-editing objectives, yielding a biased yet more sample-efficient estimator. Experiments on English$\to$Finnish, English$\to$Turkish, and English$\leftrightarrow$Chinese show consistent gains over RL baselines, and for English$\to$Turkish, performance on COMET-KIWI is comparable to advanced LLM-based systems (DeepSeek-V3.2). Our code and a set of representative pretrained models are publicly available at \url{https://github.com/NJUNLP/peg-rl} and \url{https://huggingface.co/collections/DGME/pegrl}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。