解决大模型推理中熵崩溃问题,提升推理多样性与稳定性。
Rethinking Entropy Regularization in Large Reasoning Models
- 提出SIREN方法,通过双重掩码限制探索范围。
- 在AIME24/25上实现+6.6的maj@k提升,优于以往方法。
- 适合研究强化学习与大模型推理的学者参考。
基于可验证奖励的强化学习(RLVR)在提升大推理模型(LRM)推理能力方面展现出巨大潜力,但存在熵崩溃和过早收敛的问题。传统的熵正则化方法在LRM中失效,原因在于其庞大的动作空间和长轨迹易引发全局熵爆炸。为此,本文提出SIREN(SelectIve entRopy rEgularizatioN),通过两阶段熵掩码机制——top-p掩码与峰值熵掩码——将探索限制在有意义的动作与状态子集内,并将正则化转化为自锚定形式以稳定训练。在五个数学基准测试中,SIREN在平均性能上优于先前的熵相关RLVR方法,尤其在AIME24/25上使用Qwen2.5-Math-7B模型时取得+6.6的maj@k提升。进一步分析表明,SIREN增强了响应多样性,维持了合适的熵水平,有效缓解了训练过程中验证通过率下降的过早收敛问题。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has shown great promise in enhancing the reasoning abilities of large reasoning models (LRMs). However, it suffers from a critical issue: entropy collapse and premature convergence. Naive entropy regularization, a common approach for encouraging exploration in the traditional RL literature, fails to address this problem in the context of LRM. Our analysis reveals that this failure stems from the vast action space and long trajectories in LRMs, which easily trigger a global entropy explosion as the model indiscriminately explores all possible actions and states. To address this, we propose SIREN (SelectIve entRopy rEgularizatioN), a method that confines exploration to a meaningful subset of actions and states. SIREN achieves this through a two-step entropy masking mechanism, consisting of a top-p mask and a peak-entropy mask. In addition, regularization is transformed into a self-anchored form to stabilize training. Across five mathematical benchmarks, SIREN attains superior average performance over previous entropy-related RLVR approaches, exemplified by a +6.6 maj@k improvement on AIME24/25 with Qwen2.5-Math-7B. Further analysis confirms that SIREN promotes greater response diversity and maintains entropy at an appropriate level, which helps to preserve the validation pass@k throughout training. This effectively mitigates the premature convergence problem common in RLVR for LRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。