解决大模型推理后训练导致的探索能力下降问题。
Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models
- 通过累积中间层后验分布，选择高熵深度配置作为探索候选
- 在多个基准上提升pass@1和pass@16准确率0.61和1.03个百分点
- 无需额外训练，适合增强强化学习中的高效探索
大型推理模型（LRMs）通过强化学习（RL）后训练在数学与代码推理任务中表现强劲。然而我们发现，现代推理后训练会导致意外的探索崩溃：基于温度的采样不再提升pass@$n$准确率。实验显示，后训练模型的最终层后验分布熵显著降低，而中间层熵保持较高。针对这一熵不对称现象，我们提出潜空间探索解码（LED），一种基于深度的解码策略。LED通过累积中间层后验分布，选择熵最大的深度配置作为探索候选。无需额外训练或参数，LED在多个推理基准和模型上持续提升pass@1和pass@16准确率0.61和1.03个百分点。进一步将LED融入强化学习（如使用GRPO作为滚动生成策略），可加速奖励提升并获得更高最终性能，归因于LED的高效探索能力。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have recently achieved strong mathematical and code reasoning performance through Reinforcement Learning (RL) post-training. However, we show that modern reasoning post-training induces an unintended exploration collapse: temperature-based sampling no longer increases pass@$n$ accuracy. Empirically, the final-layer posterior of post-trained LRMs exhibit sharply reduced entropy, while the entropy of intermediate layers remains relatively high. Motivated by this entropy asymmetry, we propose Latent Exploration Decoding (LED), a depth-conditioned decoding strategy. LED aggregates intermediate posteriors via cumulative sum and selects depth configurations with maximal entropy as exploration candidates. Without additional training or parameters, LED consistently improves pass@1 and pass@16 accuracy by 0.61 and 1.03 percentage points across multiple reasoning benchmarks and models. Furthermore, integrating LED into reinforcement learning, e.g., using GRPO as the rollout strategy, yields faster reward improvement and higher final performance, due to the efficient exploration capability of LED. Project page: https://github.com/AlbertTan404/LED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。