arXiv:2603.17310cs.AIcs.CL2026-03被引 2

用信息密度优化大模型推理,让思考更高效不啰嗦。

InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning

  • 通过熵轨迹分析发现优质推理具低不确定性和快速下降特征
  • 提出InfoDensity奖励机制,提升推理质量与长度效率的平衡
  • 在数学和通用推理任务中表现优于现有方法,适合追求高效推理的场景

具备长程推理能力的大语言模型常生成冗长重复的推理过程,带来不必要的计算开销。现有强化学习方法虽优化最终输出长度,却忽视中间推理步骤的质量,易导致奖励劫持。我们指出,冗余不仅是长度问题,更是中间推理质量差的表现。通过实证研究追踪大模型在推理轨迹中的每标记预测熵,发现高质量推理具有两个一致特征:不确定性收敛低、下降速度快。这表明优质推理具有信息密度高——在较短推理长度内实现低不确定性。基于此,我们提出InfoDensity奖励框架,通过熵轨迹的后缀最大包络线加长度缩放项,统一捕捉上述特性。在数学与通用推理基准上的实验表明,InfoDensity在准确率-效率权衡上优于最先进基线。

原文摘要 · Abstract (English)

Large Language Models (LLMs) with extended reasoning capabilities often generate verbose and redundant reasoning traces, incurring unnecessary computational cost. While existing reinforcement learning approaches address this by optimizing final response length, they neglect the quality of intermediate reasoning steps, leaving models vulnerable to reward hacking. We argue that verbosity is not merely a length problem, but a symptom of poor intermediate reasoning quality. To investigate this, we conduct an empirical study tracking the per-token predictive entropy of large reasoning models across reasoning trajectories. We find that high-quality reasoning traces exhibit two consistent properties: low uncertainty convergence and fast uncertainty descent. These findings suggest that high-quality reasoning traces are informationally dense, that is, reasoning steps contribute to reaching a low uncertainty level relative to the total reasoning length. Motivated by this, we propose InfoDensity, a reward framework for RL training that captures both properties through a single suffix-max envelope of the entropy trajectory, weighted by a length scaling term that favors achieving equivalent quality more concisely. Experiments on mathematical and general reasoning benchmarks demonstrate that InfoDensity outperforms state-of-the-art baselines on the accuracy-efficiency trade-off.

大模型推理强化学习信息密度效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。