用模型自生好奇感提升强化学习探索效率，解决大模型推理中的过早收敛问题。
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
- 用生成文本困惑度和价值估计方差作为内在好奇心信号
- 在AIME上比标准RLVR提升约3个百分点，避免熵崩溃
- 适合研究大模型强化学习与推理优化的读者
强化学习结合可验证奖励（RLVR）是提升大语言模型（LLM）推理能力的有效范式。然而当前的RLVR方法探索能力不足，常导致过早收敛和熵崩溃。为此，我们提出一种基于好奇心驱动探索（CDE）的框架，利用模型自身的内在好奇心引导探索。我们通过演员端的生成困惑度和批评者端多头架构的价值估计方差来形式化好奇心信号，二者均作为RLVR框架中的探索奖励。理论分析表明，演员端奖励能惩罚过度自信的错误并促进正确回答的多样性；此外，批评者端奖励与强化学习中经典的计数基探索奖励相关联。实验上，该方法在AIME基准上使用GRPO/PPO相比标准RLVR实现约+3点性能提升。进一步分析揭示了RLVR中存在校准崩溃机制，为常见大模型失效模式提供了洞见。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。