通过阶段熵控制推理长度,让大模型更简洁高效地思考。
PEAR: Phase Entropy Aware Reward for Efficient Reasoning
- 根据推理阶段的熵值动态调整奖励,区分探索与确定性阶段。
- 在4个基准上缩短响应长度,同时保持竞争力准确率。
- 无需指定长度目标,适合对效率敏感的应用场景。
大型推理模型(LRMs)在复杂推理任务中表现优异,但生成的思维链(CoT)解释常过长,冗余步骤增加推理成本并降低可用性。如何在不牺牲准确率的前提下控制输出长度仍是挑战。通过系统性实证分析,我们发现不同推理阶段的模型熵与响应长度存在一致正相关:思考阶段熵较高,反映探索性行为导致长响应;最终答案阶段熵较低,体现确定性解法。这一现象表明,不同阶段的熵可作为平衡简洁性与性能的调控变量。基于此,本文提出相位熵感知奖励(PEAR),将阶段依赖的熵纳入奖励设计,对思考阶段的过高熵进行惩罚,允许答案阶段适度探索,从而引导模型生成更简洁且能正确解题的推理路径。该方法实现响应长度的自适应控制,无需显式长度目标或刚性截断规则。在四个基准上的大量实验表明,PEAR持续减少响应长度的同时保持高准确率,且在分布外(OOD)任务中表现出强鲁棒性。代码已开源:https://github.com/iNLP-Lab/PEAR。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have achieved impressive performance on complex reasoning tasks by generating detailed chain-of-thought (CoT) explanations. However, these responses are often excessively long, containing redundant reasoning steps that inflate inference cost and reduce usability. Controlling the length of generated reasoning without sacrificing accuracy remains an open challenge. Through a systematic empirical analysis, we reveal a consistent positive correlation between model entropy and response length at different reasoning stages across diverse LRMs: the thinking phase exhibits higher entropy, reflecting exploratory behavior of longer responses, while the final answer phase shows lower entropy, indicating a more deterministic solution. This observation suggests that entropy at different reasoning stages can serve as a control knob for balancing conciseness and performance. Based on this insight, this paper introduces Phase Entropy Aware Reward (PEAR), a reward mechanism that incorporating phase-dependent entropy into the reward design. Instead of treating all tokens uniformly, PEAR penalize excessive entropy during the thinking phase and allowing moderate exploration at the final answer phase, which encourages models to generate concise reasoning traces that retain sufficient flexibility to solve the task correctly. This enables adaptive control of response length without relying on explicit length targets or rigid truncation rules. Extensive experiments across four benchmarks demonstrate that PEAR consistently reduces response length while sustaining competitive accuracy across model scales. In addition, PEAR demonstrates strong out-of-distribution (OOD) robustness beyond the training distribution. Our code is available at: https://github.com/iNLP-Lab/PEAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。