arXiv:2505.12996cs.CLcs.AI2025-05被引 13

用示例强化的强化学习,让翻译模型学会深度推理,跨语言表现更优。

ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning

  • 用强模型对比生成译文,量化差异来给奖励,提升训练效果。
  • 在11种语言上实现90个方向的翻译,文学翻译达到新顶尖水平。
  • 方法轻量易扩展,适合想提升多语种翻译质量的研究者。

近年来,大型推理模型(LRMs)如OpenAI-o1和DeepSeek-R1在数学与编程等复杂任务中表现出色。一些研究尝试将此类模型应用于神经机器翻译(MT),通过强化学习(RL)赋予模型深度推理能力。然而现有工作多集中于英语、中文等高资源语言,其他语言性能不明确;且奖励建模未充分释放强化学习潜力。本文提出一种新奖励建模方法:将策略模型的翻译结果与强模型DeepSeek-R1-671B的输出进行对比,量化差异以提供奖励。实验表明该方法显著有效。以Qwen2.5-7B-Instruct为基座,训练模型在文学翻译任务中达到新基准,优于OpenAI-o1和DeepSeek-R1。进一步扩展至11种语言,通过轻量级奖励建模,可将单一方向的强翻译能力简单迁移至90个翻译方向,实现优异的多语言翻译表现。

原文摘要 · Abstract (English)

In recent years, the emergence of large reasoning models (LRMs), such as OpenAI-o1 and DeepSeek-R1, has shown impressive capabilities in complex problems, e.g., mathematics and coding. Some pioneering studies attempt to bring the success of LRMs in neural machine translation (MT). They try to build LRMs with deep reasoning MT ability via reinforcement learning (RL). Despite some progress that has been made, these attempts generally focus on several high-resource languages, e.g., English and Chinese, leaving the performance on other languages unclear. Besides, the reward modeling methods in previous work do not fully unleash the potential of reinforcement learning in MT. In this work, we first design a new reward modeling method that compares the translation results of the policy MT model with a strong LRM (i.e., DeepSeek-R1-671B), and quantifies the comparisons to provide rewards. Experimental results demonstrate the superiority of the reward modeling method. Using Qwen2.5-7B-Instruct as the backbone, the trained model achieves the new state-of-the-art performance in literary translation, and outperforms strong LRMs including OpenAI-o1 and DeepSeeK-R1. Furthermore, we extend our method to the multilingual settings with 11 languages. With a carefully designed lightweight reward modeling in RL, we can simply transfer the strong MT ability from a single direction into multiple (i.e., 90) translation directions and achieve impressive multilingual MT performance.

机器翻译强化学习多语言推理模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。