arXiv:2505.12366cs.LGcs.AI2025-05NeurIPS被引 23

用判别式优化提升大模型推理能力,解决难度偏差与训练不稳问题。

DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization

  • 引入判别式约束优化框架,用评分函数替代群体相对目标。
  • 在六项数学推理任务上比GRPO提升7%,比DAPO提升6%。
  • 适合需要稳定训练和处理数据不平衡的模型强化场景。

DeepSeek-R1的成功使群体相对策略优化(GRPO)成为大模型推理增强的重要方法。本文分析了二元奖励设置下GRPO的目标,揭示其存在题目难度偏差的固有缺陷,并发现其与监督学习中判别方法的关联。受此启发,提出新的判别式约束优化(DisCO)框架,基于判别学习原则设计。DisCO核心改进包括:(1) 以评分函数定义的判别目标取代群体相对目标;(2) 采用非截断的强化学习代理目标作为评分函数,摒弃截断代理;(3) 使用简单有效的约束优化方法控制KL散度。实验表明,DisCO完全消除难度偏差,缓解熵不稳定问题,实现长周期稳定训练;同时可融入先进判别学习技术应对数据不平衡问题。在1.5B模型上对SFT微调模型进行数学推理能力增强,跨六项基准任务平均性能较GRPO提升7%,较DAPO提升6%。

原文摘要 · Abstract (English)

The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint. As a result, DisCO offers notable advantages over GRPO and its variants: (i) it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for a 1.5B model.

大模型推理强化学习判别学习模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。