arXiv:2505.16637cs.CLcs.AI2025-05ACL被引 14

无需参考文本,模型自评分数实现高质量机器翻译。

SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation

  • 用自评奖励机制替代外部监督,训练全程在线无依赖。
  • 仅用13K单语数据和Qwen-2.5-7B,中文-英文翻译超越多个大模型。
  • 结合外部评估工具后达到开源模型最优,甚至超越闭源模型。

大型语言模型在机器翻译领域表现突出,但现有先进模型多依赖人工标注参考文本或训练好的奖励模型,成本高且难扩展。为此,我们提出一种简单自奖励强化学习框架SSR,实现无参考、全在线的机器翻译训练,仅依赖模型自身判断奖励。使用13,000个单语例句和Qwen-2.5-7B作为主干,SSR-Zero-7B在WMT23、WMT24和Flores200基准上的中英翻译任务中,优于塔式指令模型TowerInstruct-13B、GemmaX-28-9B,以及更大的通用模型Qwen2.5-32B-Instruct。进一步通过引入COMET外部监督,最强模型SSR-X-Zero-7B在中英翻译上达到开源模型最佳性能,超越所有参数量小于720亿的开源模型,甚至优于部分闭源模型。分析表明,自奖励机制在机器翻译中优于外部大模型评分,与训练好的奖励模型结合更具互补优势。研究为自提升强化学习提供了新思路。代码、数据与模型已公开。

原文摘要 · Abstract (English)

Large language models (LLMs) have recently demonstrated remarkable capabilities in machine translation (MT). However, most advanced MT-specific LLMs heavily rely on external supervision signals during training, such as human-annotated reference data or trained reward models (RMs), which are often expensive to obtain and challenging to scale. To overcome this limitation, we propose a Simple Self-Rewarding (SSR) Reinforcement Learning (RL) framework for MT that is reference-free, fully online, and relies solely on self-judging rewards. Training with SSR using 13K monolingual examples and Qwen-2.5-7B as the backbone, our model SSR-Zero-7B outperforms existing MT-specific LLMs, e.g., TowerInstruct-13B and GemmaX-28-9B, as well as larger general LLMs like Qwen2.5-32B-Instruct in English $\leftrightarrow$ Chinese translation tasks from WMT23, WMT24, and Flores200 benchmarks. Furthermore, by augmenting SSR with external supervision from COMET, our strongest model, SSR-X-Zero-7B, achieves state-of-the-art performance in English $\leftrightarrow$ Chinese translation, surpassing all existing open-source models under 72B parameters and even outperforming closed-source models. Our analysis highlights the effectiveness of the self-rewarding mechanism compared to the external LLM-as-a-judge approach in MT and demonstrates its complementary benefits when combined with trained RMs. Our findings provide valuable insight into the potential of self-improving RL methods. We have publicly released our code, data and models.

机器翻译强化学习自奖励开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。