用尾部奖励搜索优化大模型推理,提升算力使用效率。
Predicting and improving test-time scaling laws via reward tail-guided search
- 基于奖励尾部分布预测模型表现,指导算力分配。
- 相同算力下,比传统Best-of-N策略获得更高奖励。
- 理论证明可近似最优,适合追求推理效率的开发者。
测试时扩展已成为提升大语言模型(LLM)推理能力的关键路径。尽管简单的“最佳N选一”(BoN)策略已显著提升性能,但其对N值选择、预算分配及多阶段决策缺乏合理指导,仍有巨大优化空间。现有研究虽探索了相关优化,但理论保证有限。本文提出基于尾部引导搜索的新方法,通过估计奖励尾部分布,无需全面评估即可预测LLM的缩放规律。利用该预测工具,我们设计了缩放规律引导搜索(SLG)算法,动态分配计算资源以识别并利用具有最高预测潜力的中间状态。理论上证明,SLG相较于理想信息预言机可实现可忽略的后悔值,并在期望奖励上达到原本需多项式更大算力才能实现的效果。实验验证了该框架在不同LLM与奖励模型上的有效性,表明尾部引导分配在相同算力下始终优于BoN。代码已开源。
原文摘要 · Abstract (English)
Test-time scaling has emerged as a critical avenue for enhancing the reasoning capabilities of Large Language Models (LLMs). Though the straight-forward ''best-of-$N$'' (BoN) strategy has already demonstrated significant improvements in performance, it lacks principled guidance on the choice of $N$, budget allocation, and multi-stage decision-making, thereby leaving substantial room for optimization. While many works have explored such optimization, rigorous theoretical guarantees remain limited. In this work, we propose new methodologies to predict and improve scaling properties via tail-guided search. By estimating the tail distribution of rewards, our method predicts the scaling law of LLMs without the need for exhaustive evaluations. Leveraging this prediction tool, we introduce Scaling-Law Guided (SLG) Search, a new test-time algorithm that dynamically allocates compute to identify and exploit intermediate states with the highest predicted potential. We theoretically prove that SLG achieves vanishing regret compared to perfect-information oracles, and achieves expected rewards that would otherwise require a polynomially larger compute budget required when using BoN. Empirically, we validate our framework across different LLMs and reward models, confirming that tail-guided allocation consistently achieves higher reward yields than Best-of-$N$ under identical compute budgets. Our code is available at https://github.com/PotatoJnny/Scaling-Law-Guided-search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。