解析大模型推理时计算量扩展的原理，揭示如何通过采样优化输出质量。
Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
- 用贝叶斯线性回归建模大模型作为裁判的推理过程，引入温度调节的软最大选择机制。
- 当奖励与教师模型接近时，增加采样数k可单调降低泛化误差，最优误差随1/k²衰减。
- 发现推理计算优势随任务难度上升而减弱，且存在最佳采样温度与最优采样数限制。
近期大语言模型的发展表明，将计算资源从训练阶段转向推理阶段具有显著优势。然而，推理阶段扩展的原理尚不明确。本文提出一个可解析分析的推理阶段扩展模型：基于奖励加权采样的贝叶斯线性回归，其中奖励由线性模型决定，用于模拟大模型作为裁判的场景。在高维设定下，我们推导出后验预测均值与方差的闭式表达。当训练数据来自教师模型时，分析泛化误差表现。通过采样k次推理结果，并以温度调节的二次奖励进行softmax选择。若奖励与教师模型差异不大，泛化误差随采样数k单调下降；但最优奖励通常不同于教师模型。当奖励严重失配时，存在有限最优的k，超过后误差反而上升。对固定k，存在最佳采样温度。实验验证了这些结论在使用大模型作裁判的推理中成立。在‘最佳-k’极限下，若以教师模型为奖励，理论证明泛化误差衰减为Θ(1/k²)，并利用极值理论确定首项系数。这些公式明确了在何种条件下扩大推理计算优于收集更多数据。最后，发现任务难度增加会削弱推理计算的优势。
原文摘要 · Abstract (English)
Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time. However, the principles behind inference time scaling are not well understood. In this paper, we introduce an analytically tractable model of inference-time scaling: Bayesian linear regression with a reward-weighted sampler, where the reward is determined from a linear model, modeling LLM-as-a-judge scenario. We study this problem in the high-dimensional regime, where the deterministic equivalents dictate a closed-form expression for the posterior predictive mean and variance. We analyze the generalization error when training data are sampled from a teacher model. We draw $k$ inference-time samples and select via softmax at a temperature applied to a quadratic reward. When the reward is not too different from the teacher, the generalization error decreases monotonically with increasing inference time samples $k$. However, the specific reward that optimizes inference-time selection generally differs from the teacher. In contrast, substantial reward misspecification induces a finite optimal $k$ beyond which more sampling can increase the generalization error. For fixed $k$, there exists an optimal sampling temperature. We experimentally verify these facts in large language model inference with an additional large language model as a judge. In the "best-of-$k$" limit with the teacher as reward, we theoretically show that the generalization error decays as $Θ(1/k^2)$ and determine the leading coefficient via extreme value theory. These formulas delineate domains where scaling inference-time computation is provably preferable to collecting more data. Finally, we demonstrate that when task difficulty increases, the previously mentioned advantage of inference-time compute degrades.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。