用模型自身生成的角度信号,让强化学习训练快2.5倍。
Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals
- 根据隐藏层向量角度集中度动态选数据,替代随机采样。
- 在数学与编程任务上训练效率提升超2.5倍,半量数据超基线。
- 无需额外标注,适合想加速大模型微调的研究者。
当前大型语言模型的强化微调范式因均匀采样导致重复查询,存在样本效率低的问题。已有工作尝试通过启发式难度指标进行课程学习,但忽略了模型自身产生的内在学习信号,造成训练效果不佳。本文发现一种模型内生信号——角度集中度,能有效反映模型对特定数据的学习能力。理论与实证均表明,隐藏状态向量的角度分布与梯度之间存在关联,模型更倾向于从角度集中度高的数据中学习。基于此,我们提出GAIN-RL(Gradient-driven Angle-Informed Navigated RL)框架,利用模型自身的角度集中信号,在每轮训练中动态选择最具影响力的样本,确保持续有效的梯度更新,显著提升训练效率。实验显示,GAIN-RL(GRPO)在多样化的数学与编码任务及不同模型规模下,训练效率提升超过2.5倍;且其高效采样策略实现数据高效训练,在仅使用原数据一半的情况下,性能超越全量数据下的原始GRPO。代码已开源:https://github.com/wangqinsi1/GAINRL/tree/main。
原文摘要 · Abstract (English)
Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models (LLMs) suffer from sample inefficiency due to the redundant exposure of identical queries under uniform data sampling. While previous work has explored curriculum learning via heuristic difficulty metrics, these strategies exhibit limitations by neglecting the intrinsic learning signals generated by the model itself, thus leading to suboptimal training regimes. In this paper, we identify a model-inherent signal termed angle concentration that effectively reflects an LLM's capacity to learn from specific data. We theoretically and empirically demonstrate a correlation between the angular distribution of token hidden state vectors and the resulting gradient, revealing a learning preference for data exhibiting higher angle concentration. Inspired by this finding, we propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. By leveraging the model's intrinsic angle concentration signal, GAIN-RL dynamically selects training data in each epoch, ensuring consistently impactful gradient updates and thus significantly enhancing overall training efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5x acceleration in training efficiency across diverse mathematical and coding tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient sampling yields data-efficient training, achieving better performance with half the original data compared to vanilla GRPO with full training data. Code is realsed at https://github.com/wangqinsi1/GAINRL/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。