arXiv:2510.03199cs.LGstat.ML2025-10被引 4

提出最优推理策略BoM,提升大模型在复杂任务中的准确率。

Best-of-Majority: Minimax-Optimal Strategy for Pass@$k$ Inference Scaling

  • 结合多数投票与Best-of-N优点,先筛选高频答案再选最优
  • 理论证明其误差随k增大呈平方根下降,达到最优性能极限
  • 实验显示在数学题上优于传统方法,且不因样本量增加而退化

大模型推理常生成一组候选答案并用多数投票或Best-of-N策略选择。面对困难任务,单次选择表现不佳,因此评估常用Pass@$k$:允许提交最多$k$个响应,仅使用其中最优者计算失败率。本文研究更一般的Pass@$k$推理设置,证明多数投票和Best-of-N均无法实现理想缩放性能。为此,提出新的推理策略Best-of-Majority(BoM),关键步骤是在$N$次采样中先筛选高频率候选答案,再选取前$k$个最优奖励。理论证明当采样预算$N=\tildeΩ(C^*)$时,BoM的后悔值为$O(ε_{\mathrm{opt}}+\sqrt{ε_{\mathrm{RM}}^2C^*/k})$,其中$C^*$为覆盖率系数,$ε_{\mathrm{RM}}$为奖励模型估计误差,$ε_{\mathrm{opt}}$为最优响应奖励估计误差。进一步建立匹配下界,证明该算法为极小极大最优。实验表明,在数学问题上的推理中,BoM优于多数投票和Best-of-N。

原文摘要 · Abstract (English)

LLM inference often generates a batch of candidates for a prompt and selects one via strategies like majority voting or Best-of- N (BoN). For difficult tasks, this single-shot selection often underperforms. Consequently, evaluations commonly report Pass@$k$: the agent may submit up to $k$ responses, and only the best of them is used when computing regret. Motivated by this, we study inference scaling in the more general Pass@$k$ inference setting, and prove that neither majority voting nor BoN exhibits the desirable scaling with $k$ and the sampling budget $N$. Combining the advantages of majority voting and BoN, we propose a new inference strategy called Best-of-Majority (BoM), with a pivotal step that restricts the candidates to the responses with high frequency in the $N$ samples before selecting the top-$k$ rewards. We prove that when the sampling budget is $N=\tildeΩ(C^*)$, the regret of BoM is $O(ε_{\mathrm{opt}}+\sqrt{ε_{\mathrm{RM}}^2C^*/k})$, where $C^*$ is the coverage coefficient, $ε_{\mathrm{RM}}$ is the estimation error of the reward model, and $ε_{\mathrm{opt}}$ is the estimation error of reward at the optimal response. We further establish a matching lower bound, certifying that our algorithm is minimax optimal. Beyond optimality, BoM has a key advantage: unlike majority voting and BoN, its performance does not degrade when increasing $N$. Experimental results of inference on math problems show BoM outperforming both majority voting and BoN.

大模型推理策略优化Pass@k

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。