arXiv:2506.05850cs.CLcs.AI2025-06

多语言大模型推理时会不自觉用母语,影响跨语言能力。

Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models

  • 用强化学习让模型做长串推理,发现越深入推理越倾向用英语
  • 数学题越难、解码越随机,越容易出现语言退化现象
  • 通过奖励机制或调整模型可缓解问题,但性能与语言一致性难两全

基于可验证奖励的强化学习(RLVR)在训练大语言模型(LLM)进行长链推理(CoT)时表现优异。然而我们系统研究发现:多语言模型在非英语提示下,其推理过程会逐渐回归预训练主导语言(如英语),称为跨语言退化。由于长链推理放大了语言先验的影响,模型在提升推理深度与保持目标语言一致性之间存在未被充分理解的权衡。我们在多个数学数据集的翻译版本上使用组相对策略优化(GRPO)训练模型,并追踪任务准确率与推理链的语言一致性。实验显示:(i) RLVR训练中,模型推理语言随性能提升而系统性偏向主语言;(ii) 英语先验、长链优化、任务难度和高熵解码共同加剧该现象,且不限于数学领域;(iii) 引入语言一致性奖励、解码控制或更均衡的模型结构可缓解退化,但始终存在性能与语言保真度之间的权衡。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable reward (RLVR) has been instrumental in eliciting strong reasoning capabilities from large language models (LLMs) via long chains of thought (CoT). During RLVR training, we formalize and systemically study an empirical phenomenon whereby a multilingual model's CoT reverts to its dominant pre-training language (e.g., English) even when prompted in another language, which we term Cross-lingual Collapse. Because the long-CoT regime magnifies exposure to linguistic priors, the underlying trade-off between maximizing reasoning depth and preserving target-language fidelity has remained under-characterized. To examine this trade-off, we train LLMs with Group-Relative Policy Optimization (GRPO) on translated versions of math datasets widely used to elicit long-CoT reasoning. Throughout training, we track both task accuracy and the language consistency of reasoning chains. Our experiments yield three findings: (i) under RLVR, CoT in LLMs systematically drifts toward the pre-training dominant language as reasoning performance rises; (ii) English-centric priors, long-CoT GRPO optimization, task difficulty, and high-entropy decoding jointly amplify this drift, and the pattern persists beyond mathematics; and (iii) interventions that favor target-language traces--via a language-consistency reward, decoding-time controls, or more balanced backbones--mitigate collapse but reveal a persistent performance-fidelity trade-off.

大模型推理多语言强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。