通过熵趋势奖励，让大模型推理更短更准
ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning
- 用熵变化趋势替代固定长度惩罚，引导推理逐步降低不确定性
- 在4个基准上使推理长度减少67%，准确率提升9.9%
- 适合追求高效精准推理的开发者和研究者
链式思考（CoT）能提升大模型在复杂任务上的表现，但常产生过长且低效的推理过程。现有方法通过长度惩罚或全局熵减来缩短推理，隐含假设全程低不确定性为佳。本文指出，推理效率取决于不确定性的变化轨迹：具有明显下降熵趋势的推理路径显著更短。基于此，提出熵趋势奖励（ETR），一种关注轨迹的优化目标，鼓励逐步降低不确定性，同时允许有限局部探索。将ETR集成至组相对策略优化（GRPO）框架，在多个推理模型和挑战性基准上评估。结果表明，ETR始终实现更优的准确性-效率权衡，使DeepSeek-R1-Distill-7B在四个基准上准确率提升9.9%，推理长度平均减少67%。代码已开源。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning improves large language model performance on complex tasks, but often produces excessively long and inefficient reasoning traces. Existing methods shorten CoTs using length penalties or global entropy reduction, implicitly assuming that low uncertainty is desirable throughout reasoning. We show instead that reasoning efficiency is governed by the trajectory of uncertainty. CoTs with dominant downward entropy trends are substantially shorter. Motivated by this insight, we propose Entropy Trend Reward (ETR), a trajectory-aware objective that encourages progressive uncertainty reduction while allowing limited local exploration. We integrate ETR into Group Relative Policy Optimization (GRPO) and evaluate it across multiple reasoning models and challenging benchmarks. ETR consistently achieves a superior accuracy-efficiency tradeoff, improving DeepSeek-R1-Distill-7B by 9.9% in accuracy while reducing CoT length by 67% across four benchmarks. Code is available at https://github.com/Xuan1030/ETR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。