通过动态选题提升推理模型强化学习训练速度
SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
- 根据题目难易度动态选择训练样本，避免无效计算
- 实测训练速度提升2至6倍，精度不下降
- 无需调参，可直接接入现有强化学习框架
使用强化学习（RL）训练大语言模型时，若采用可验证的奖励信号，能显著提升其推理能力，但传统均匀采样方式效率低下。本文提出SPEED方法，一种基于难度估计的自适应在线课程学习机制，优先选择中等难度的提示样本以最大化学习效率。理论上，中等难度提示能提升梯度估计器的信噪比，加速收敛。实验表明，该方法实现2至6倍的训练加速，且不降低模型精度，无需手动调参，可无缝集成到标准强化学习算法中。
原文摘要 · Abstract (English)
Training large language models with reinforcement learning (RL) against verifiable rewards significantly enhances their reasoning abilities, yet remains computationally expensive due to inefficient uniform prompt sampling. We introduce Selective Prompting with Efficient Estimation of Difficulty (SPEED), an adaptive online RL curriculum that selectively chooses training examples of intermediate difficulty to maximize learning efficiency. Theoretically, we establish that intermediate-difficulty prompts improve the gradient estimator's signal-to-noise ratio, accelerating convergence. Empirically, our efficient implementation leads to 2x to 6x faster training without degrading accuracy, requires no manual tuning, and integrates seamlessly into standard RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。