用投票法提升大模型答案选择准确率，比传统方法更稳。
Majority of the Bests: Improving Best-of-N via Bootstrapping
- 通过自举法估算最佳输出分布，选众数而非单个最优。
- 在30组实验中25组超越传统方法，提升显著。
- 适合追求高可靠性的生成任务，尤其奖励模型不准时。
从大语言模型采样多个输出并选择最频繁（自一致）或得分最高（Best-of-N）的候选，是提升离散答案任务准确率的常用方法。当奖励模型完美时，Best-of-N常达近满分表现；但面对不完美的奖励模型，其性能急剧下降。本文分析了BoN输出分布，发现正确答案虽未必概率接近1，却常为最可能结果。基于此，提出新机制Majority-of-the-Bests（MoB），通过自举法估计分布并选取众数。跨五个基准、三种基础模型、两种奖励模型的实验显示，MoB在30组设置中有25组优于BoN。同时提供自举法的一致性理论支持。MoB是比BoN和自一致性更简单且更强的替代方案，也推动了更精细选择机制的研究。
原文摘要 · Abstract (English)
Sampling multiple outputs from a Large Language Model (LLM) and selecting the most frequent (Self-consistency) or highest-scoring (Best-of-N) candidate is a popular approach to achieve higher accuracy in tasks with discrete final answers. Best-of-N (BoN) selects the output with the highest reward, and with perfect rewards, it often achieves near-perfect accuracy. With imperfect rewards from reward models, however, BoN fails to reliably find the correct answer and its performance degrades drastically. We consider the distribution of BoN's outputs and highlight that, although the correct answer does not usually have a probability close to one under imperfect rewards, it is often the most likely outcome. This suggests that the mode of this distribution can be more reliably correct than a sample from it. Based on this idea, we propose Majority-of-the-Bests (MoB), a novel selection mechanism that estimates the output distribution of BoN via bootstrapping and selects its mode. Experimental results across five benchmarks, three different base LLMs, and two reward models demonstrate consistent improvements over BoN in 25 out of 30 setups. We also provide theoretical results for the consistency of the bootstrapping. MoB serves as a simple, yet strong alternative to BoN and self-consistency, and more broadly, motivates further research in more nuanced selection mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。