通过解耦探索与利用,提升大模型推理能力。
DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
- 将奖励调节拆分为两个独立超参数,精准控制探索与利用平衡。
- 在逻辑推理数据集上达96.9%准确率,显著优于现有方法。
- 适用于多种强化学习框架,适合研究大模型推理优化者。
推理规模的扩大进一步推动大语言模型向通用人工智能迈进,大规模强化学习可激发长链思维推理。当前多数推理方法依赖手工设计的奖励函数,但强化学习中探索与利用的权衡涉及多重复杂因素,手动设计奖励函数的理论与实证影响尚未充分探讨。本文提出解耦组奖励优化(DGRO),一种通用的大语言模型推理强化学习算法。一方面,将传统正则化系数解耦为两个独立超参数:一个调节策略梯度项,另一个控制采样策略距离。该解耦不仅实现探索与利用的精确控制,还可无缝扩展至Kimi k1.5中的在线策略镜像下降(OPMD)及直接奖励优化。另一方面,我们发现奖励方差显著影响收敛速度与最终性能。通过理论分析与大量实证验证,包括详细的消融实验,结果表明DGRO在逻辑推理数据集上平均准确率达96.9%,并在多个数学基准上展现强泛化能力。
原文摘要 · Abstract (English)
Inference scaling further accelerates Large Language Models (LLMs) toward Artificial General Intelligence (AGI), with large-scale Reinforcement Learning (RL) to unleash long Chain-of-Thought reasoning. Most contemporary reasoning approaches usually rely on handcrafted rule-based reward functions. However, the tarde-offs of exploration and exploitation in RL algorithms involves multiple complex considerations, and the theoretical and empirical impacts of manually designed reward functions remain insufficiently explored. In this paper, we propose Decoupled Group Reward Optimization (DGRO), a general RL algorithm for LLM reasoning. On the one hand, DGRO decouples the traditional regularization coefficient into two independent hyperparameters: one scales the policy gradient term, and the other regulates the distance from the sampling policy. This decoupling not only enables precise control over balancing exploration and exploitation, but also can be seamlessly extended to Online Policy Mirror Descent (OPMD) algorithms in Kimi k1.5 and Direct Reward Optimization. On the other hand, we observe that reward variance significantly affects both convergence speed and final model performance. We conduct both theoretical analysis and extensive empirical validation to assess DGRO, including a detailed ablation study that investigates its performance and optimization dynamics. Experimental results show that DGRO achieves state-of-the-art performance on the Logic dataset with an average accuracy of 96.9\%, and demonstrates strong generalization across mathematical benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。