让大模型自动选语言生成多语回复，提升跨语言训练效果
Learning to Route Languages for Multilingual Policy Optimization
- 用可学习的路由机制动态选择训练时使用的语言
- 在固定生成预算下，多语言回溯使性能显著提升
- 适合需要多语言对齐的强化学习场景
大型语言模型在异构多语言语料上训练，但现有策略优化方法通常将每个训练问题限制为单一响应语言，或依赖固定主导语言进行监督。我们提出语言路由策略优化（LRPO），一种在线策略优化框架，将语言视为可选变量。LRPO 为每个训练问题生成多语言回溯，并将它们的相对质量整合进基于偏好策略更新中，在固定回溯预算下提升了训练信号的多样性和信息量。为自适应决定强化学习中探索哪些语言，我们引入一个可训练的语言路由器，形式化为多臂赌博机，平衡对低使用语言的探索与对更优语言的利用。大量实验表明，LRPO 持续提升多语言性能，证明自适应语言路由能有效实现跨语言知识利用。相关资源已开源：https://github.com/Guochry/LRPO。
原文摘要 · Abstract (English)
Large language models~(LLMs) are trained on heterogeneous multilingual corpora, yet existing policy optimization methods often implicitly restrict each training question to a single response language or rely on a fixed dominant language for supervision. We propose language-routed policy optimization (LRPO), an online policy optimization framework that treats language as a selectable variable. LRPO elicits multilingual rollouts for each training question and integrates their relative quality into preference-based policy updates, increasing the diversity and informativeness of training signals under the fixed rollout budget. To adaptively determine which languages to explore during reinforcement learning, we introduce a trainable language router formulated as a multi-armed bandit, balancing exploration of underutilized languages with exploitation of more informative ones. Extensive experiments show that LRPO consistently improves multilingual performance, demonstrating that adaptive language routing enables effective cross-lingual knowledge exploitation for training. We release all the resources at https://github.com/Guochry/LRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。