发现大模型强化学习更新有可预测的线性规律，能大幅加速训练。
On Predictability of Reinforcement Learning Dynamics for Large Language Models
- 发现参数更新的主成分几乎决定推理提升，且随训练线性变化。
- 仅用早期训练数据即可准确预测最终效果，性能损失小于4%。
- 提出AlphaRL框架，提速最高2.5倍，无需额外模块或调参。
大语言模型（LLM）推理能力的提升主要依赖强化学习（RL），但其训练过程中的参数动态仍不清晰。本文揭示了两类基本性质：（1）秩-1主导性，即参数更新矩阵的主奇异子空间几乎完全决定推理性能提升，可恢复超过99%的性能增益；（2）秩-1线性动态，该主导子空间在训练中呈线性演化，使从早期检查点即可准确预测最终结果。在8个LLM和7种算法上的实验验证了这些性质的普适性。更重要的是，基于此提出AlphaRL加速框架，通过短时早期训练窗口外推最终参数更新，在保持超过96%推理性能的同时实现最高2.5倍加速，无需额外模块或超参数调整。该发现为大规模强化学习提供了通用、实用的工具，推动可解释、高效的大模型训练范式发展。
原文摘要 · Abstract (English)
Recent advances in reasoning capabilities of large language models (LLMs) are largely driven by reinforcement learning (RL), yet the underlying parameter dynamics during RL training remain poorly understood. This work identifies two fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1 Dominance, where the top singular subspace of the parameter update matrix nearly fully determines reasoning improvements, recovering over 99\% of performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace evolves linearly throughout training, enabling accurate prediction from early checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the generalizability of these properties. More importantly, based on these findings, we propose AlphaRL, a plug-in acceleration framework that extrapolates the final parameter update using a short early training window, achieving up to 2.5 speedup while retaining \textgreater 96\% of reasoning performance without extra modules or hyperparameter tuning. This positions our finding as a versatile and practical tool for large-scale RL, opening a path toward principled, interpretable, and efficient training paradigm for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。