用训练数据中常见答案做基线，发现重复采样未必比猜答案更有效。
Keep Guessing? When Considering Inference Scaling, Mind the Baselines
- 基于训练数据频次枚举常见答案，作为推理基线
- 某些模型在数学和知识题上，枚举法比反复采样效果更好
- 揭示了重复采样提升覆盖率的真实上限，适合评估模型真实能力
通过多次采样增加大语言模型推理计算量，可持续提高问题求解覆盖率（解决的问题比例）。我们推测这种提升部分源于标准评测基准的答案分布偏向少数常见答案。为此，我们定义了一种基线方法：根据训练集中答案的出现频率进行枚举。在数学推理和事实知识两个领域进行实验发现，该基线在部分模型上优于重复采样；对其他模型，其表现与仅用10次模型采样、剩余答案通过枚举获取的混合策略相当。该基线使我们能更准确地衡量重复采样在去提示依赖猜测背景下的实际增益。
原文摘要 · Abstract (English)
Scaling inference compute in large language models (LLMs) through repeated sampling consistently increases the coverage (fraction of problems solved) as the number of samples increases. We conjecture that this observed improvement is partially due to the answer distribution of standard evaluation benchmarks, which is skewed towards a relatively small set of common answers. To test this conjecture, we define a baseline that enumerates answers according to their prevalence in the training set. Experiments spanning two domains -- mathematical reasoning and factual knowledge -- reveal that this baseline outperforms repeated model sampling for some LLMs, while the coverage for others is on par with that of a mixture strategy that obtains $k$ answers by using only $10$ model samples and similarly guessing the remaining $k-10$ attempts via enumeration. Our baseline enables a more accurate measurement of how much repeated sampling improves coverage in such settings beyond prompt-agnostic guessing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。