arXiv:2511.08567cs.LGcs.AI2025-11中稿 · NeurIPS被引 47

揭示大模型强化学习中参数更新的隐藏规律,打破稀疏性假象。

The Path Not Taken: RLVR Provably Learns Off the Principals

  • 提出三门理论解释为何更新集中在少数参数,实为优化偏置所致。
  • 发现RLVR通过非主方向微调实现性能提升,谱漂移极小且方向对齐。
  • 警告直接套用SFT的高效微调方法会失效,需设计适配强化学习的新算法。

强化学习与可验证奖励(RLVR)能有效提升大语言模型的推理能力,但看似仅更新少量参数。我们重新审视这一悖论,发现稀疏性是模型依赖优化偏置的表象:固定预训练模型时,参数更新始终集中于特定区域,跨运行高度一致,且不受数据集和强化学习策略影响。我们提出三门理论:门Ⅰ(KL锚点)施加KL约束更新;门Ⅱ(模型几何)引导梯度偏离主方向,进入低曲率、保持谱结构的子空间;门Ⅲ(精度)将微小更新隐藏于非偏好区域,使偏置表现似稀疏。我们验证该理论,并首次提供参数级的RLVR学习动态刻画:RLVR在权重空间中沿非主方向学习,以极小谱漂移、减少主子空间旋转及更新对齐实现增益。相比之下,SFT聚焦主权重,破坏谱结构,甚至落后于RLVR。这些结果首次揭示了RLVR在参数空间中的清晰规律,表明其处于与SFT截然不同的优化范式,因此直接复用SFT时代的参数高效微调(PEFT)方法可能出错,如我们在先进稀疏微调与LoRA变体上的案例所示。本研究旨在推动对RLVR的白盒理解,促进面向几何感知、专为RLVR设计的新算法开发,而非复用旧有启发式方法。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parameters. We revisit this paradox and show that sparsity is a surface artifact of a model-conditioned optimization bias: for a fixed pretrained model, updates consistently localize to preferred parameter regions, highly consistent across runs and largely invariant to datasets and RL recipes. We mechanistically explain these dynamics with a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained update; Gate II (Model Geometry) steers the step off principal directions into low-curvature, spectrum-preserving subspaces; and Gate III (Precision) hides micro-updates in non-preferred regions, making the off-principal bias appear as sparsity. We then validate this theory and, for the first time, provide a parameter-level characterization of RLVR's learning dynamics: RLVR learns off principal directions in weight space, achieving gains via minimal spectral drift, reduced principal-subspace rotation, and off-principal update alignment. In contrast, SFT targets principal weights, distorts the spectrum, and even lags RLVR. Together, these results provide the first parameter-space account of RLVR's training dynamics, revealing clear regularities in how parameters evolve. Crucially, we show that RL operates in a distinct optimization regime from SFT, so directly adapting SFT-era parameter-efficient fine-tuning (PEFT) methods can be flawed, as evidenced by our case studies on advanced sparse fine-tuning and LoRA variants. We hope this work charts a path toward a white-box understanding of RLVR and the design of geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics.

强化学习参数效率模型优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。