发现并修复了大模型强化学习中的负梯度陷阱，提升推理能力。
On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
- 通过分析错误响应的统一惩罚机制，揭示了正确答案概率下降的根源。
- 提出NTHR方法，动态降低导致性能下降的错误词元惩罚强度。
- 在0.5B至3B参数模型上均实现稳定性能提升，适合改进数学推理系统。
强化学习已广泛用于提升大语言模型的推理能力，其中组相对策略优化（GRPO）成为主流算法。然而我们发现一种此前未被识别的现象——懒惰似然位移（LLD），即正确回答的似然值在训练中几乎不变甚至下降。这一现象与最近发现的直接偏好优化（DPO）中的错位问题类似，归因于负梯度的影响。本文对GRPO的学习动态进行理论分析，指出LLD的根源在于对所有错误响应中的词元施加相同强度的惩罚。为此，我们提出NTHR方法，通过降低对导致LLD的词元的惩罚权重来缓解该问题。不同于以往基于DPO的方法，NTHR利用了GRPO的组结构，以正确回答作为锚点识别关键词元。在数学推理基准上的实验表明，NTHR有效缓解了LLD，在0.5B至3B参数的多个模型上均获得持续性能提升。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging as a widely used algorithm in recent systems. Despite GRPO's widespread adoption, we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training. This behavior mirrors a recently discovered misalignment issue in Direct Preference Optimization (DPO), attributed to the influence of negative gradients. We provide a theoretical analysis of GRPO's learning dynamic, identifying the source of LLD as the naive penalization of all tokens in incorrect responses with the same strength. To address this, we develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens. Experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD, yielding consistent performance gains across models ranging from 0.5B to 3B parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。