用熵聚类的时空结构做内在奖励,提升大模型测试时推理质量。
Entropy Centroids as Intrinsic Rewards for Test-Time Scaling

- 通过高熵段落的时序分布定义熵质心,捕捉模型不确定性模式。
- 在14B到480B模型上,最低熵质心法持续优于基线,且随模型增大收益更稳定。
- 无需外部奖励模型,适合大规模语言模型测试时计算扩展场景。
大规模语言模型测试时计算扩展的有效方法是生成多个响应并选择最优者,如Grok Heavy和Gemini Deep Think。现有选择方法多依赖需训练的外部奖励模型,带来额外计算开销。作为替代,先前研究探索了置信度、熵等内在信号,但其聚合方式噪声较大。本文观察到推理过程中高熵词元倾向于形成连续片段,提供了比单个词元更稳定的不确定性表征。这些片段揭示了推理过程中的不确定性时序模式。受此启发,我们提出利用不确定性时序结构作为内在奖励。首先,将段级不确定性基本单元形式化为高熵段(HEP),即从高熵词元开始,至连续低熵词元出现时结束的可变长度段。随后,借鉴物理中质心概念,定义熵质心为所有HEP沿生成轨迹的加权平均位置。直观上,较低的熵质心意味着早期探索后快速进入自信生成阶段,我们发现这常对应更高响应质量。基于此,提出最低熵质心方法:从多个候选响应中选取熵质心最低者。在数学、代码生成、逻辑推理及代理任务上,跨14B至480B模型规模的实验表明,该方法始终优于现有基线,且模型越大,增益越稳定。代码已开源:https://github.com/hkust-nlp/entropy-centroid。
原文摘要 · Abstract (English)
An effective way to scale up test-time compute of large language models is to sample multiple responses and then select the best one, as in Grok Heavy and Gemini Deep Think. Existing selection methods often rely on external reward models, which requires training a strong reward model and introduces additional computation overhead. As an alternative, previous approaches have explored intrinsic signals, such as confidence and entropy, but these signals are noisy with naive aggregation. In this work, we observe that high-entropy tokens tend to cluster into consecutive groups during inference, providing a more stable notion of model uncertainty than individual tokens. Together, these clusters reveal temporal patterns of model uncertainty throughout the inference process. Motivated by this observation, we propose to use the temporal structure of uncertainty as an intrinsic reward. To this end, we first formalize the basic unit of segment-level uncertainty as the High Entropy Phase (HEP), a variable-length segment that begins at a high-entropy token and ends when consecutive low-entropy tokens appear. We then define the Entropy Centroid, inspired by the concept of the center of mass in physics, as the weighted average position of all HEPs along the trajectory. Intuitively, a lower centroid indicates early exploration followed by confident generation, which we find often corresponds to higher response quality. Based on this insight, we propose the Lowest Centroid method, which selects the response with the lowest entropy centroid among multiple candidates. Experiments on mathematics, code generation, logical reasoning, and agentic tasks, across model scales ranging from 14B to 480B, show that Lowest Centroid consistently outperforms existing baselines and delivers stable gains as model size increases. Code is available at https://github.com/hkust-nlp/entropy-centroid.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。