让大模型先选对的,再选多样的,提升推理准确性。
Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
- 按正确性而非置信度决定采样,避免盲目探索
- 在数学和通用推理任务上显著提升准确率
- 适合需要高质量推理的科研与工程应用
大语言模型在复杂推理任务中越来越重要,通常需要生成多种思维链来探索不同解法。这涉及两个矛盾目标:在不确定步骤增加随机性以促进探索,同时确保每条路径的准确性和质量。现有方法通过提高温度或扩大候选词集增强探索,或在生成后剔除低置信度样本以保证可靠性,但这些做法混淆了不同来源的不确定性。本文提出应以正确性为优先,而非仅依赖置信度。我们建议在预期正确性高的位置采样,而在正确性低的位置减少采样。为此提出三种策略:Greedy-Threshold 在极低置信度步骤采用贪婪选择;Calibrated-TopK 和 Calibrated-epsilon 根据分等级的正确性估计设定截断阈值。实验表明,这些方法在数学推理(如MATH)和通用推理(如GSM8K)基准上均取得显著提升,挑战了当前基于置信度的解码范式。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly applied to complex tasks that require extended reasoning. In such settings, models often benefit from diverse chains-of-thought to arrive at multiple candidate solutions. This requires two competing objectives: to inject enough stochasticity to explore multiple reasoning chains, and to ensure sufficient accuracy and quality in each path. Existing works pursue the first objective by increasing exploration at highly uncertain steps with higher temperature or larger candidate token sets, while others improve reliability by rejecting samples with low confidence post-generation, implying that low confidence correlates with low answer quality. These two lines of thought are in conflict, as they conflate different sources of uncertainty. To resolve this, we argue that the decoding rule should be calibrated by correctness, not confidence alone. We should sample from tokens with higher estimated correctness, and reduce sampling where expected correctness is low. We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps. Calibrated-TopK and Calibrated-epsilon set truncation threshold based on estimated rank-wise correctness. Together, our findings challenge prevailing heuristics about decoding under uncertainty and show gains across math and general reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。