arXiv:2504.10160cs.CLcs.AI2025-04EMNLP被引 64

用强化学习让大模型自动生成高质量翻译,无需人工标注。

MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning

  • 设计混合奖励机制,让模型通过自我推理优化翻译质量。
  • 在中英翻译上超越多个基线模型,7B版本达到62.25分,接近GPT-4o水平。
  • 适用于多语言和低资源场景,适合研究强化学习与机器翻译的结合。

大规模强化学习已显著提升大模型在数学与编程等可验证任务中的推理能力,但在输出灵活、难以自动评估的机器翻译(MT)领域仍缺乏探索。本文提出 MT-R1-Zero,首个无需监督微调或冷启动的 R1-Zero 强化学习框架开源实现。通过规则-指标混合奖励机制,引导模型通过涌现推理提升翻译质量。在 WMT 24 英中基准上,MT-R1-Zero-3B-Mix 平均领先 TowerInstruct-7B-v0.2 1.26 分;MT-R1-Zero-7B-Mix 在所有指标上平均得分为 62.25,媲美 GPT-4o 和 Claude-3.5-Sonnet 等先进闭源模型,其中 MT-R1-Zero-7B-Sem 在语义指标上达到当前最优。该方法在分布外任务、多语言及低资源设置下表现出强泛化能力。对不同初始化与奖励机制的深入分析揭示了奖励设计、模型适应性、训练动态与涌现推理模式的关键作用。代码已开源:https://github.com/fzp0424/MT-R1-Zero。

原文摘要 · Abstract (English)

Large-scale reinforcement learning (RL) methods have proven highly effective in enhancing the reasoning abilities of large language models (LLMs), particularly for tasks with verifiable solutions such as mathematics and coding. However, applying this idea to machine translation (MT), where outputs are flexibly formatted and difficult to automatically evaluate with explicit rules, remains underexplored. In this work, we introduce MT-R1-Zero, the first open-source adaptation of the R1-Zero RL framework for MT without supervised fine-tuning or cold-start. We propose a rule-metric mixed reward mechanism to guide LLMs towards improved translation quality via emergent reasoning. On the WMT 24 English-Chinese benchmark, our MT-R1-Zero-3B-Mix achieves competitive performance, surpassing TowerInstruct-7B-v0.2 by an average of 1.26 points. Meanwhile, our MT-R1-Zero-7B-Mix attains a high average score of 62.25 across all metrics, placing it on par with advanced proprietary models such as GPT-4o and Claude-3.5-Sonnet, while the MT-R1-Zero-7B-Sem variant achieves state-of-the-art scores on semantic metrics. Moreover, our work exhibits strong generalization capabilities on out-of-distribution MT tasks, robustly supporting multilingual and low-resource settings. Extensive analysis of model behavior across different initializations and reward metrics offers pioneering insight into the critical role of reward design, LLM adaptability, training dynamics, and emergent reasoning patterns within the R1-Zero paradigm for MT. Our code is available at https://github.com/fzp0424/MT-R1-Zero.

机器翻译强化学习大模型自推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。