arXiv:2602.00983cs.CLcs.AI2026-02中稿 · the 29th Internati…被引 1

提出DISPO算法,让大模型数学推理强化学习更高效稳定。

DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning

  • 分离正确与错误回答的权重裁剪,控制策略更新方式。
  • 在AIME'24上达61.04%准确率,优于CISPO和DAPO。
  • 适合追求高效且稳定训练的大模型数学推理研究者。

基于可验证奖励的强化学习已成为提升大语言模型数学推理能力的有前景范式。当前方法存在明显权衡:PPO类方法(如GRPO/DAPO)训练稳定但学习缓慢,因策略更新受信任区域约束;而REINFORCE类方法(如CISPO)虽效率更高,却因裁剪重要性采样权重仍允许域外梯度,导致性能不稳定。为此,我们提出DISPO——一种简单有效的REINFORCE类算法,将正确与错误响应的重要性采样权重分别上、下裁剪,形成四种可控的策略更新模式。通过针对性消融实验发现:对正确响应,权重>1提升平均词元熵(探索),<1降低熵(提炼),二者均有益但过度使用会引发性能渐进下降;对错误响应,过度限制裁剪会导致突发性能崩溃——权重>1引发重复输出,<1则导致响应长度消失。通过独立调节四组裁剪参数,DISPO维持探索与提炼平衡,避免灾难性失败,在AIME'24上达到61.04%准确率(优于CISPO的55.42%和DAPO的50.21%),并在多个基准与模型上实现类似提升。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models particularly in mathematics. Current approaches in this domain present a clear trade-off: PPO-style methods (e.g., GRPO/DAPO) offer training stability but exhibit slow learning trajectories due to their trust-region constraints on policy updates, while REINFORCE-style approaches (e.g., CISPO) demonstrate improved learning efficiency but suffer from performance instability as they clip importance sampling weights while still permitting non-zero gradients outside the trust-region. To address these limitations, we introduce DISPO, a simple yet effective REINFORCE-style algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes. Through targeted ablations, we uncover how each regime impacts training: for correct responses, weights >1 increase the average token entropy (i.e., exploration) while weights <1 decrease it (i.e., distillation) -- both beneficial but causing gradual performance degradation when excessive. For incorrect responses, overly restrictive clipping triggers sudden performance collapse through repetitive outputs (when weights >1) or vanishing response lengths (when weights <1). By separately tuning these four clipping parameters, DISPO maintains the exploration-distillation balance while preventing catastrophic failures, achieving 61.04% on AIME'24 (vs. 55.42% CISPO and 50.21% DAPO) with similar gains across various benchmarks and models.

强化学习数学推理大模型训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。