用单语文本训练模型，提升低资源语言翻译能力
Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation
- 仅用单语数据构建奖励信号，避免依赖稀缺双语数据
- 在101种语言上训练，1400个方向超越最强开源模型
- 针对质量评估模型漏洞设计修复机制，防止奖励劫持
大型语言模型（LLMs）在高资源语言对上的机器翻译表现优异，但在低资源语言上仍存在差距。现有后训练方法严重依赖高质量双语数据，而这些数据对低资源语言往往匮乏或缺失。本文提出WALAR，一种仅使用单语文本的强化学习训练方法，可在不损害高资源语言性能的前提下，显著提升模型在大量低资源语言上的翻译能力。核心洞察是：现有基于源端的多语言质量评估（QE）模型存在失效模式（或称“漏洞”），利用此类模型进行强化学习会放大这些漏洞，导致模型性能下降。为此，我们引入词对齐与语言对齐技术，有效缓解了WALAR中强化学习的奖励漏洞。通过持续训练支持101种语言的LLM，实验表明该模型在Flores-101数据集的1400个语言方向上，显著优于当前最强的开源多语言模型LLaMAX。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capability in machine translation on high-resource language pairs, yet their performance on low-resource translation still lags behind. Existing post-training methods rely heavily on high-quality parallel data, which are often scarce or unavailable for low-resource languages. In this paper, we introduce WALAR, a reinforcement training method using only monolingual text to elevate LLMs' translation capabilities on massive low-resource languages while retaining their performance on high-resource languages. Our key insight is based on the observation of failure modes (or "holes") in existing source-based multilingual quality estimation (QE) models. Reinforcement learning (RL) using these QE models tends to amplify such holes, resulting in poorer multilingual LLMs. We develop techniques including word alignment and language alignment to mitigate such holes in WALAR's reward for RL training. We continually trained an LLM supporting translation of 101 languages using WALAR. The experiments show that our new model outperforms LLaMAX, one of the strongest open-source multilingual LLMs by a large margin on 1400 language directions on Flores-101 dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。