arXiv:2607.09693cs.LGcs.AI2026-07

不训练模型,仅靠内部熵变化规律就能大幅提升大模型推理能力。

Depth-Entropy Guided Sampling for Training-Free LLM Reasoning

论文配图:Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
图 1 · 摘自论文原文
  • 利用模型各层输出熵的演变模式作为质量信号,指导采样。
  • 在多个推理任务上达到当前无需训练方法的最高准确率,尤其在困难样本上优势明显。
  • 适合追求高效推理、无须标注数据或训练的场景,如部署阶段优化。

强化学习已成为提升大语言模型推理能力的主流范式,但需要昂贵的训练、精心构造的数据和奖励信号。近期研究发现,仅在测试时对基础模型输出进行锐化采样,即可恢复大部分强化学习带来的增益,但现有方法仅依赖输出层似然,忽略了Transformer内部前向传播动态。本文提出深度熵引导采样(DEGS),一种无需训练、仅在测试时使用的新型方法,利用层间熵塌缩作为内在质量信号。我们观察到更强的推理模型——包括经过强化学习微调的版本——表现出显著的“后期塌缩”特征:在深层网络之前,logit-lens解码的熵值持续较高,随后收敛。我们定义了每条输入序列的塌缩深度 $D(oldsymbol{x})$,并构建联合目标 $π(oldsymbol{x}) /propto p(oldsymbol{x})^α ext{exp}(βD(oldsymbol{x}))$,将其嵌入MCMC幂采样框架中(DEGS-MCMC)。在三个开源模型和四个推理基准上,该方法以近乎零成本的单个候选信号,在采样轨迹中累积出当前最优的无需训练精度。性能提升最显著的是跨域任务与更难的样本子集,正是传统似然信号失效之处,且仅带来单数字百分比级别的时钟时间开销。尽管在数学题上略逊于专为该任务训练的GRPO模型,但在非域任务(如GPQA)上,三款模型均超越其表现,且全程无需训练、奖励模型或标注数据。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals. Recent work shows that sampling from sharpened base-model distributions at test time recovers much of the RL gain, yet existing methods rely solely on output-layer likelihoods and ignore the transformer's internal forward-pass dynamics. We introduce Depth-Entropy Guided Sampling (DEGS), a training-free, test-time method that exploits layer-wise entropy collapse as an intrinsic quality signal. We observe that stronger reasoners -- including RL-posttrained variants -- exhibit a distinctive "late collapse": logit-lens decoded entropy stays elevated until deeper layers before converging. We define a per-sequence collapse depth $D(\mathbf{x})$ and a joint objective $π(\mathbf{x}) \propto p(\mathbf{x})^α\exp(βD(\mathbf{x}))$ that combines sequence likelihood with this depth-entropy structure, instantiated inside an MCMC power-sampling framework (DEGS-MCMC). Across three open-weight models and four reasoning benchmarks, this near-chance per-candidate signal compounds over the sampling trajectory into state-of-the-art training-free accuracy, with gains largest out of domain and on the harder splits -- exactly where likelihood alone falls short -- at single-digit-percent wall-clock overhead. DEGS narrowly trails an in-house GRPO reference on the math splits GRPO was trained for, yet surpasses it out of domain on GPQA for all three models, without any training, reward model, or labeled data.

大模型推理采样优化无训练熵分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。