通过检测推理过程熵值变化,实现大模型高效早停,节省12%-22%算力。
Entropy After </Think> for reasoning model early exiting
- 在推理后监测熵值变化,动态判断是否可提前停止
- 在MATH500和AIME2025上减少12%-22% token使用量
- 适用于无梯度访问的黑盒场景,兼容小模型代理
推理型大模型虽随思维链增长性能提升,但常出现过度思考现象,即在得出正确答案后仍持续修正。我们从分布动态角度量化验证:在大量推理轨迹中,模型在早期已能稳定输出正确答案,后续推理浪费资源。为此提出新信号“</Think>后熵”(EAT),在添加停止思考标记后,监测后续词元的熵变化。当通过指数移动平均阈值判断熵值方差趋于平稳时,即为合理退出时机。该方法支持按需分配算力,相比固定预算更高效。实证显示,在MATH500和AIME2025上,EAT可减少12%-22%的token消耗且不降低准确率。该方法亦适用于黑盒环境,通过小模型代理计算EAT——我们已验证其可行性:使用1.5B模型对Llama 70B进行早停,以及用4B本地模型对Claude 3.7进行控制。
原文摘要 · Abstract (English)
Reasoning LLMs show improved performance with longer chains of thought. However, recent work has highlighted their tendency to overthink, continuing to revise answers even after reaching the correct solution. We quantitatively confirm this inefficiency from the distribution dynamics perspective by tracking Pass@1 for answers averaged over a large number of rollouts and find the model often begins to always produce the correct answer early in the reasoning, making extra reasoning tokens wasteful. To detect and prevent overthinking, we propose a simple and inexpensive novel signal, Entropy After </Think> (EAT), for monitoring and deciding whether to exit reasoning early. By appending a stop thinking token (</think>) and monitoring the entropy of the following token as the model reasons, we obtain a trajectory that decreases and stabilizes when Pass@1 plateaus; thresholding its variance under an exponential moving average yields a practical stopping rule. Importantly, our approach enables adaptively allocating compute based on the EAT trajectory, allowing us to spend compute in a more efficient way compared with fixing the token budget for all questions. Empirically, on MATH500 and AIME2025, EAT reduces token usage by 12 - 22% without harming accuracy. EAT also remains effective in black box settings where logits from the reasoning model are not accessible, and EAT is computed with proxy models: We verified the feasibility via early stopping Llama 70B with a 1.5B model and Claude 3.7 with a local 4B model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。