针对大模型推理中熵坍缩问题，提出位置感知的熵校准方法
PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR
- 基于局部熵和候选竞争构建软掩码，动态识别关键决策位置
- 在五项数学推理任务上超越强基线，尤其在AIME类任务提升显著
- 适合需要长程推理与高可靠性的强化学习场景
基于可验证奖励的强化学习（RLVR）能提升大语言模型的推理能力，但常面临策略熵快速坍缩问题，即策略过早聚焦于少数高概率推理路径。全局熵正则化虽能促进探索，但在长推理轨迹中对所有位置均匀增加熵效率低下，因多数位置并非决策相关。本文提出位置感知熵校准（PAEC），一种逐标记熵管理框架：通过局部top-p熵与前两名候选竞争构建软掩码，并施加基于锚点的下界惩罚，防止选中位置熵坍缩。在五个数学推理基准上的实验表明，PAEC在宏平均多数投票性能上优于强基线，尤其在AIME风格任务上表现突出。结果表明，推理强化学习中的熵管理应定位为对决策敏感位置的选择性探索分配，而非均匀注入随机性。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths. While global entropy regularization can encourage exploration, uniformly increasing entropy across all token positions is inefficient for long reasoning trajectories, where many tokens are not decision-relevant. We propose Position-Aware Entropy Calibration (PAEC), a token-level entropy-management framework that constructs a soft mask from local top-p entropy and top-two candidate competition, and applies an anchor-based lower-bound penalty to prevent selected-position entropy collapse. Experiments on five mathematical reasoning benchmarks show that PAEC improves macro-average majority-vote performance over strong RLVR baselines, with clear gains on AIME-style tasks. Our results suggest that entropy management in reasoning RL should be formulated as selective exploration allocation over decision-sensitive positions rather than uniform randomness injection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。