用无参考强化学习提升多语言大模型翻译质量
Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation
- 基于自动生成评分，用无参考强化学习优化翻译模型
- 46种语言上超越现有开源模型，接近商业系统水平
- 适合想改进多语言翻译的开发者和研究者
我们研究了使用开放的大语言模型进行无参考后训练以提升多语言机器翻译性能。从监督微调后的MiLMMT-46-v0.1模型出发，采用组相对策略优化（GRPO），奖励函数融合两个无参考质量评估模型并受语言识别门控。随后对监督微调与强化学习模型检查点进行线性插值，得到MiLMMT-46-v1.0。在46种语言上，该模型持续优于原始监督微调版本，超过多个近期强开源基线（如Seed-X、HY-MT2、TranslateGemma），并在无参考评估中达到领先于谷歌翻译、Gemini 3 Pro和GPT-5等专有系统的水平。进一步研究发现，在线策略蒸馏（OPD）可达到但无法超越强化学习插值所达的质量上限。模型与代码已公开，以促进后续研究。
原文摘要 · Abstract (English)
We study reference-free post-training for multilingual machine translation with open large language models. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, we apply Group Relative Policy Optimization (GRPO) with a reward that averages two reference-free quality estimation models and is gated by language identification. We then linearly interpolate the supervised fine-tuning (SFT) and reinforcement learning (RL) model checkpoints to obtain MiLMMT-46-v1.0. Across 46 languages, the resulting models consistently improve translation quality over their SFT counterparts, outperform strong recent open baselines, including Seed-X, HY-MT2, and TranslateGemma, and achieve leading reference-free scores against evaluated proprietary systems such as Google Translate, Gemini 3 Pro, and GPT-5. We further investigate on-policy distillation (OPD) and find that it reaches, but does not surpass, the quality frontier achieved by RL with checkpoint interpolation. We release the models and code to facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。