动态分配推理路径并调节梯度,提升大模型推理的效率与稳定性。
How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization
- 按问题差异动态分配推理路径,减少梯度方差。
- 通过梯度感知调节优势值,避免正确动作梯度衰减。
- 适合需要高效训练大模型推理能力的研究者使用。
基于可验证奖励的强化学习(RLVR)在大语言模型(LLM)推理中表现优异,但现有方法存在资源分配不合理与策略优化动态失衡的问题:(i) 均匀分配推理路径忽略了不同问题间梯度方差的差异性;(ii) softmax策略结构导致高置信度正确动作的梯度衰减,而过度更新可能引发训练不稳。为此,我们提出理论驱动的双管齐下优化框架DynaMO。在序列层面,证明均匀分配次优,并从第一性原理推导出最小方差分配方案,以伯努利方差作为梯度信息量的可计算代理。在词元层面,基于梯度幅度边界分析,设计梯度感知的优势调节机制,补偿高置信度正确动作的梯度衰减,并利用熵变作为可计算指标稳定过大的更新幅度。在多个数学推理基准上的大量实验表明,相比强基线,DynaMO持续取得性能提升。代码已开源:https://github.com/GithubX-F/DynaMO-RL。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for Large Language Model (LLM) reasoning, yet current methods face key challenges in resource allocation and policy optimization dynamics: (i) uniform rollout allocation ignores gradient variance heterogeneity across problems, and (ii) the softmax policy structure causes gradient attenuation for high-confidence correct actions, while excessive gradient updates may destabilize training. Therefore, we propose DynaMO, a theoretically-grounded dual-pronged optimization framework. At the sequence level, we prove that uniform allocation is suboptimal and derive variance-minimizing allocation from the first principle, establishing Bernoulli variance as a computable proxy for gradient informativeness. At the token level, we develop gradient-aware advantage modulation grounded in theoretical analysis of gradient magnitude bounds. Our framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. Extensive experiments conducted on a diverse range of mathematical reasoning benchmarks demonstrate consistent improvements over strong RLVR baselines. Our implementation is available at: https://github.com/GithubX-F/DynaMO-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。