用熵值识别关键决策点,提升采样效率,让模型更聪明地重试推理路径。
Reasoning with Sampling: Cutting at Decision Points
- 通过令牌熵判断推理中的关键决策点,精准重采样。
- 在多个评测集上超越基线和强化学习训练模型。
- 适合想低成本提升推理能力的研究者与工程师。
前沿推理模型通常通过强化学习对基础语言模型进行后训练生成。近期研究发现,仅通过对基础模型分布进行锐化(即幂分布)采样,即可获得媲美训练模型的推理能力,无需额外训练、标注数据或验证器。然而,高效从幂分布采样仍具挑战,需采样器能有效“混合”至目标分布,即在不同推理策略间切换。现有方法随机均匀选取推理轨迹中的某位置作为“切点”并重采样后续内容,但这类切点常位于局部细节而非关键决策点。本文提出熵切分梅特罗波利斯-黑斯廷斯算法(Entropy-Cut Metropolis-Hastings),利用基础模型的下一个词熵作为代理信号,识别关键决策点并从这些位置重采样。实证表明熵突增是决策点的有效指示;在简化推理模型中,理论证明该方法的混合时间仅随推理决策数增长,而非随令牌总数增长(后者可大得多)。在MATH500、HumanEval、GPQA Diamond和AIME26上,本方法持续优于基线及强化学习训练模型。
原文摘要 · Abstract (English)
Frontier reasoning models are produced by posttraining base language models with reinforcement learning. Recent work has challenged this by showing that sampling from a sharpened version of the base model's distribution, a so-called power distribution, elicits comparable reasoning without additional training, curated datasets, or verifiers. However, making this method practical requires efficiently sampling from the power distribution. A sampler needs to "mix" to the power distribution, which necessitates moving between modes of the target distribution; intuitively, e.g., trying different reasoning strategies. The samplers proposed in prior works repeatedly select a "cut" position in the current reasoning trace uniformly at random and resample the suffix from that position onward. However, reasoning traces typically contain a few consequential decisions (e.g., the choice of proof strategy or algorithm), and we observe that a uniformly chosen cut tends to rewrite local details rather than revisit decision points. We introduce an algorithm (Entropy-Cut Metropolis-Hastings) that uses the base model's next-token entropy as a proxy to identify key decision points and resample from those positions. We empirically verify that entropy jumps are a useful proxy for decision points and, in a stylized model of reasoning, prove that our method's mixing time scales with the number of decisions in a trace rather than with the number of tokens, which can be much larger. Across MATH500, HumanEval, GPQA Diamond, and AIME26, our method consistently improves over baselines and RL-trained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。