通过动态调整生成参数,让模型用更少推理次数做出更准判断。
CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning
- 先探索再校准:分两阶段优化生成路径,提升推理效率。
- 在数学题上减少4倍推理次数仍达同等准确率,固定预算下更优。
- 适配多种生成策略,适合追求高精度推理的开发者使用。
推理时增加计算资源(测试时扩展)能提升语言模型性能,尤其在推理任务中。然而,流行的 Best-of-N 采样方法随着采样数 $N$ 增大,收益递减。为此,我们提出一个通用的测试时校准框架,无需重训练大模型即可自适应引导模型向高回报推理路径靠拢,并在有限采样条件下理论上提升期望奖励下界。在此框架下,我们提出 CarBoN(Calibrated Best-of-N),一种两阶段方法:首先探索解空间,再通过输入相关的温度 $T$ 和加性偏移向量 $δ$ 校准对数概率,引导生成更可靠的推理结果。在 MATH-500 与 AIME-2024 数据集上的实验表明,CarBoN 显著提升效率,最多减少 4 倍推理轮次即可达到相同准确率,且在固定预算下常取得更高准确率。我们还分析了 $T$ 与 $δ$ 在平衡输出多样性和正确性中的互补作用,并证明该框架可推广至如束搜索等步骤级采样策略。
原文摘要 · Abstract (English)
Allocating more computation during inference time (test-time scaling) improves language model performance, especially for reasoning tasks. However, popular methods like Best-of-$N$ sampling often show diminishing returns as $N$ increases. To address this inefficiency, we introduce a general test-time calibration framework that adaptively modifies the model toward high-reward reasoning paths, with theoretical guarantees of improving the lower bound of expected reward under finite sampling, all without large language model (LLM) retraining. Within this framework, we propose CarBoN (Calibrated Best-of-$N$), a two-phase method that first explores the solution space and then learns a calibration of the logits via an input-specific temperature $T$ and additive shift vector $δ$, guiding generation toward more reliable reasoning. Experiments on MATH-500 and AIME-2024 show that CarBoN improves efficiency, with up to $4\times$ fewer rollouts to reach the same accuracy, while often achieving higher accuracy under fixed budgets. We also analyze the complementary roles of $T$ and $δ$ in balancing output diversity and correctness, and demonstrate that the framework also generalizes to step-level sampling strategies such as beam search. For more information, please refer to our project page at huggingface.co/spaces/TrustSafeAI/Test-Time-Calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。