解决大模型推理训练中的梯度冲突问题,提升长链条推理能力。
DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization
- 通过细粒度打分动态屏蔽相似样本对,从源头消除梯度冲突。
- 引入高质量锚点增强困难任务的训练信号,提升样本效率。
- 在9个数学与分布外泛化任务上达到新最优,平均准确率提升4.7%。
大语言模型的发展推动了从表面指令遵循向长期推理能力的范式转变。尽管组相对策略优化(GRPO)因卓越性能成为激发此类推理能力的关键机制,但仍面临训练不稳定和样本效率低下的问题。我们从理论上识别出根源在于在线回溯中样本缺乏区分性:对于常规查询,高度同质的样本引发破坏性梯度冲突;而对于困难查询,有效正样本稀缺导致优化无效。为此,我们提出具有区分性感知的组相对策略优化(DaGRPO)。DaGRPO包含两个核心机制:(1) 序列级梯度修正,利用细粒度评分动态掩蔽区分度低的样本对,从源头消除梯度冲突;(2) 离线数据增强,引入高质量锚点以恢复困难任务的训练信号。在9个数学推理与分布外(OOD)泛化基准上的大量实验表明,DaGRPO显著优于现有SFT、GRPO及混合基线,在数学基准上平均准确率提升4.7%。深入分析证实,该方法有效缓解梯度爆炸,并加速长链推理能力的涌现。
原文摘要 · Abstract (English)
The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift from superficial instruction following to rigorous long-horizon reasoning. While Group Relative Policy Optimization (GRPO) has emerged as a pivotal mechanism for eliciting such post-training reasoning capabilities due to its exceptional performance, it remains plagued by significant training instability and poor sample efficiency. We theoretically identify the root cause of these issues as the lack of distinctiveness within on-policy rollouts: for routine queries, highly homogeneous samples induce destructive gradient conflicts; whereas for hard queries, the scarcity of valid positive samples results in ineffective optimization. To bridge this gap, we propose Distinctiveness-aware Group Relative Policy Optimization (DaGRPO). DaGRPO incorporates two core mechanisms: (1) Sequence-level Gradient Rectification, which utilizes fine-grained scoring to dynamically mask sample pairs with low distinctiveness, thereby eradicating gradient conflicts at the source; and (2) Off-policy Data Augmentation, which introduces high-quality anchors to recover training signals for challenging tasks. Extensive experiments across 9 mathematical reasoning and out-of-distribution (OOD) generalization benchmarks demonstrate that DaGRPO significantly surpasses existing SFT, GRPO, and hybrid baselines, achieving new state-of-the-art performance (e.g., a +4.7% average accuracy gain on math benchmarks). Furthermore, in-depth analysis confirms that DaGRPO effectively mitigates gradient explosion and accelerates the emergence of long-chain reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。