针对语言模型推理优化强化学习算法，显著提升准确率并减少训练时间。
Effective Reinforcement Learning for Reasoning in Language Models
- 采用基于PPO的在线策略算法，避免传统离线策略偏差。
- 去除KL散度约束后生成更简洁且准确率更高。
- 提出DASH算法，训练提速83%适用于小模型高效训练。
强化学习（RL）已成为提升语言模型（LM）在数学与编程等推理任务中能力的有前景方法。然而，多数现代RL算法最初为机器人控制设计，与语言模型推理需求差异显著。本文针对小模型在计算资源受限下的推理能力提升，分析了各类算法设计选择对准确率与计算效率的影响。研究发现：（i）在线策略RL显著优于监督微调（SFT）；（ii）基于PPO的离线策略更新反而提升了准确率而非降低方差；（iii）移除KL散度约束可生成更简洁文本并提高准确率。此外，我们发现推理与反向传播最优批大小不一致是主要性能瓶颈。为此提出DASH算法：通过预采样大批次并以小增量累积梯度，结合低优势样本过滤机制。实验表明，相较标准GRPO实现，DASH将训练时间减少83%且不损失准确率。本研究为语言模型推理中的有效强化学习算法设计提供了关键洞见。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a promising strategy for improving the reasoning capabilities of language models (LMs) in domains such as mathematics and coding. However, most modern RL algorithms were designed to target robotics applications, which differ significantly from LM reasoning. We analyze RL algorithm design decisions for LM reasoning, for both accuracy and computational efficiency, focusing on relatively small models due to computational constraints. Our findings are: (i) on-policy RL significantly outperforms supervised fine-tuning (SFT), (ii) PPO-based off-policy updates increase accuracy instead of reduce variance, and (iii) removing KL divergence can lead to more concise generations and higher accuracy. Furthermore, we find that a key bottleneck to computational efficiency is that the optimal batch sizes for inference and backpropagation are different. We propose a novel algorithm, DASH, that performs preemptive sampling (i.e., sample a large batch and accumulate gradient updates in small increments), and gradient filtering (i.e., drop samples with small advantage estimates). We show that DASH reduces training time by 83% compared to a standard implementation of GRPO without sacrificing accuracy. Our findings provide valuable insights on designing effective RL algorithms for LM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。