提出新方法稳定控制大模型强化学习中的探索熵,避免过早收敛。
SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
- 基于温度自适应正样本设计熵控制正则项
- 在多个任务上显著提升通过率,最优探索水平非单调
- 适合需要持续推理能力的大型语言模型后训练
强化学习是大语言模型后训练的关键范式,但广泛使用的组相对策略优化(GRPO)常出现熵崩溃:探索迅速消失,策略过早收敛,样本多样性下降,影响训练效果。现有修复方法如熵奖励和截断机制,难以维持稳定的探索水平,常导致熵振荡或奖励下降。本文发现熵动态中被忽视的不对称性:高温采样下,正样本促进熵增长,负样本抑制熵。理论分析表明,熵下降时,正样本更新对温度的导数严格为正,说明高温正样本可对抗熵衰减,延缓甚至逆转熵崩溃。受此启发,提出SCOPE-RL,通过温度自适应正样本构建正则项实现稳定定量的熵控制。大量实验显示,SCOPE-RL在Pass@1和Pass@$k$上均优于强基线。结果表明,摆脱熵崩溃可提升推理性能,且收益具有非单调性,存在最优探索水平。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a key paradigm for post-training large language models (LLMs), but the widely used Group Relative Policy Optimization (GRPO) often suffers from entropy collapse: exploration quickly disappears, policies converge prematurely, and sample diversity declines, ultimately harming training effectiveness. Existing remedies, including entropy bonuses and clip-based methods, rarely keep entropy within a stable exploration regime and often introduce oscillatory entropy or reward degradation. In this work, we identify a previously overlooked asymmetry in entropy dynamics: under high-temperature sampling, positive and negative samples have opposite effects on policy entropy. Specifically, high-temperature positive samples promote entropy growth, whereas negative samples suppress it. We provide a theoretical explanation for this phenomenon: when entropy decreases during policy updates, its derivative with respect to temperature is strictly positive under positive-sample updates, indicating that high-temperature positive samples can counteract entropy decay, thereby slowing entropy collapse and potentially reversing it. Motivated by this insight, we propose SCOPE-RL, a stable and quantitative entropy control framework through a regularization term constructed from temperature-adaptive positive samples. Extensive experiments show that SCOPE-RL consistently outperforms strong RL baselines on both Pass@1 and Pass@$k$. Our results provide evidence that escaping entropy collapse can improve reasoning performance, while also showing that the benefit is non-monotonic, with an optimal level of exploration for RL post-training in reasoning LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。