发现大模型强化学习训练中存在稳定的线性规律,可大幅提升训练效率。
Linear Dynamics in the RLVR Training of Large Language Models
- 通过轨迹分析发现参数与输出概率在训练中呈高度线性变化(R² > 0.7)。
- 基于线性结构的权重外推使训练提速6.1倍,输出外推提升4.2%平均性能。
- 适合关注高效训练、模型稳定性与推理能力优化的研究者。
强化学习结合可验证奖励(RLVR)显著提升了面向推理的大语言模型性能,但其内部训练动态仍不清晰。本文对RLVR进行系统性轨迹级分析,发现无论模型家族、算法或配置如何,训练均进入稳定线性阶段:参数权重与输出对数概率在教师强制评估下呈现高度线性演化(R² > 0.7)。通过控制实验与理论分析,我们证明该线性源于高方差、噪声强的训练信号,如同低通滤波器,使优化集中在低维稳定漂移路径上。更重要的是,该线性结构不仅是描述性的,更是预测性和可操作的:权重空间外推在保持标准强化学习性能的同时实现6.1倍训练加速;输出空间外推作为轻量干预,有效缓解后期模型崩溃,在数学与编程基准上平均提升4.2%性能。代码已开源。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has driven significant performance gains in reasoning-oriented large language models (LLMs), yet its internal training dynamics remain largely a black box. In this work, we perform a comprehensive trajectory-level analysis of RLVR and uncover a striking regularity: across various model families, RL algorithms, and training configurations, RLVR consistently enters a robust linear regime, where both parameter weights and output log-probabilities, measured rigorously via teacher-forced evaluation, evolve in a highly linear manner ($R^2 > 0.7$). Through controlled experiments and theoretical analysis, we demonstrate that this linearity is not a coincidence, but stems from the high-variance, noisy nature of RLVR training signals, which act as a low-pass filter to concentrate optimization along a stable, low-dimensional drift. Moreover, we show that this linear structure is not merely descriptive but powerfully predictive and actionable. Specifically, weight-space extrapolation matches the performance of standard RL optimization while achieving a 6.1x training speedup through periodic re-grounding. Meanwhile, output-space extrapolation serves as a lightweight intervention that effectively bypasses late-stage model collapse, consistently outperforming standard RL across mathematical and coding benchmarks, with an average performance improvement of 4.2%. Our code is available at https://github.com/Miaow-Lab/RLVR-Linearity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。