用英语专家模型做桥梁,让多语言大模型学会可靠推理。
Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model
- 用英语强模型生成参考答案,通过语义匹配奖励多语言模型。
- 在多个评测集上使多语言模型平均性能提升16.41%和10.17%。
- 无需目标语言标注数据,适合资源匮乏语言的推理能力提升。
尽管强化学习提升了大型语言模型(LLMs)的推理能力,但这些进步主要局限于英语,导致不同语言间存在显著性能差距。为解决这一问题,我们提出基于语义可验证奖励的桥接强化学习(PB-RLSVR),通过不依赖目标语言人工标注数据来增强多语言推理能力。该方法利用高性能英语模型作为“桥接”模型生成推理任务的参考答案,再根据多语言模型输出与英语参考答案的语义等价性进行奖励,从而将桥接模型的推理能力跨语言迁移。我们评估了多种跨语言语义奖励函数,包括基于嵌入和机器翻译的方法。在一系列多语言推理基准上的实验表明,该方法显著缩小了英语与其他语言之间的性能差距,大幅优于传统PPO基线。具体而言,该框架使Llama-3.1-8B-Instruct和Qwen3-32B的平均多语言性能分别提升16.41%和10.17%,展示了构建真正多语言推理智能体的一种高效且数据节约的新路径。
原文摘要 · Abstract (English)
While reinforcement learning has advanced the reasoning abilities of Large Language Models (LLMs), these gains are largely confined to English, creating a significant performance disparity across languages. To address this, we introduce Pivot-Based Reinforcement Learning with Semantically Verifiable Rewards (PB-RLSVR), a novel framework that enhances multilingual reasoning by circumventing the need for human-annotated data in target languages. Our approach employs a high-performing English LLM as a "pivot" model to generate reference responses for reasoning tasks. A multilingual model is then rewarded based on the semantic equivalence of its responses to the English reference, effectively transferring the pivot model's reasoning capabilities across languages. We investigate several cross-lingual semantic reward functions, including those based on embeddings and machine translation. Extensive experiments on a suite of multilingual reasoning benchmarks show that our method significantly narrows the performance gap between English and other languages, substantially outperforming traditional PPO baselines. Specifically, our PB-RLSVR framework improves the average multilingual performance of Llama-3.1-8B-Instruct and Qwen3-32B by 16.41% and 10.17%, respectively, demonstrating a powerful and data-efficient approach to building truly multilingual reasoning agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。