利用语言不平衡自动奖励，让大模型自我提升多语言能力。
Language Imbalance Driven Rewarding for Multilingual Self-improving
- 用语言使用频率差异作为奖励信号驱动模型迭代优化
- 非主流语言性能提升7.46%，主流语言也同步改善13.9%
- 适合想提升多语言模型泛化能力的研究者
大语言模型在众多任务中已达到顶尖水平，但其进展主要惠及英语、中文等“主流语言”，其他语言仍被严重忽视。这种语言不均衡现象虽限制了应用广度，却为模型自提升提供了天然的优先级信号。本文提出语言不平衡驱动奖励机制，将模型内部主导语言与非主导语言之间的使用差距转化为奖励信号。通过迭代式直接偏好优化（DPO）训练，该方法不仅提升了非主导语言的表现，还增强了主导语言的能力，形成正向反馈循环。在Meta-Llama-3-8B-Instruct上进行两轮迭代后，在指令遵循和算术推理任务中持续提升：X-AlpacaEval基准平均胜率提高7.46%，MGSM基准准确率提升13.9%。本工作为多语言模型自进化提供初步探索路径，代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks. However, these advancements have predominantly benefited "first-class" languages such as English and Chinese, leaving many other languages underrepresented. This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportunity to bootstrap the multilingual capabilities of LLM in a self-improving manner. Thus, we propose $\textit{Language Imbalance Driven Rewarding}$, where the inherent imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal. Iterative DPO training demonstrates that this approach not only enhances LLM performance in non-dominant languages but also improves the dominant language's capacity, thereby yielding an iterative reward signal. Fine-tuning Meta-Llama-3-8B-Instruct over two iterations of this approach results in continuous improvements in multilingual performance across instruction-following and arithmetic reasoning tasks, evidenced by an average improvement of 7.46% win rate on the X-AlpacaEval leaderboard and 13.9% accuracy on the MGSM benchmark. This work serves as an initial exploration, paving the way for multilingual self-improvement of LLMs. The code is available at https://github.com/ZNLP/Language-Imbalance-Driven-Rewarding
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。