用信息论方法优化大模型推理效率，减少冗余思考。
Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens
- 引入信息偏差与增益指标，量化推理路径优劣。
- 新策略使推理耗 token 减少 50.8%，准确率提升 1.10%。
- 适合追求高效推理的模型部署与应用开发者。
大型推理模型（LRMs）虽显著提升了多步推理能力，但常伴随过长的推理链条。本文从信息论视角重新审视推理效率，揭示推理长度与语义效率之间的根本权衡。提出两个指标：InfoBias 用于衡量偏离理想推理路径的程度，InfoGain 用于量化每一步的信息贡献。实证分析显示，较长的推理链通常具有更高的信息偏差和递减的信息增益，尤其在错误答案中更为明显。基于此，我们提出一种基于熵的自适应思考策略，当置信度足够高时动态终止推理。相较于默认的 Vanilla Think 模式，在 QwQ-32B 上六项涵盖不同推理类型和难度的任务中，该策略实现平均准确率提升 1.10%，令牌使用量降低 50.80%，展现出更优的效率与推理性能。结果表明，基于熵的方法在提升大语言模型的准确性与成本效率方面具有巨大潜力。
原文摘要 · Abstract (English)
The recent rise of Large Reasoning Models (LRMs) has significantly improved multi-step reasoning performance, but often at the cost of generating excessively long reasoning chains. This paper revisits the efficiency of such reasoning processes through an information-theoretic lens, revealing a fundamental trade-off between reasoning length and semantic efficiency. We propose two metrics, InfoBias and InfoGain, to quantify divergence from ideal reasoning paths and stepwise information contribution, respectively. Empirical analyses show that longer reasoning chains tend to exhibit higher information bias and diminishing information gain, especially for incorrect answers. Motivated by these findings, we introduce an entropy-based Adaptive Think strategy that dynamically halts reasoning once confidence is sufficiently high, improving efficiency while maintaining competitive accuracy. Compared to the Vanilla Think approach (default mode), our strategy yields a 1.10% improvement in average accuracy and a 50.80% reduction in token usage on QwQ-32B across six benchmark tasks spanning diverse reasoning types and difficulty levels, demonstrating superior efficiency and reasoning performance. These results underscore the promise of entropy-based methods for enhancing both accuracy and cost-effiiciency in large language model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。