用精确贝叶斯基准测试大模型是否真会算法推理,发现多数模型其实只是猜答案。
Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal reference
- 构建可穷举的8600万条程序库,给出严格贝叶斯最优参考答案
- 675B模型最高答对92%,但45个分布都比随机键入还差
- 揭示模型推理能力不随规模提升,且会自我破坏已解决任务
衡量大语言模型是否具备算法推理能力一直困难,因多数基准仅提供答案而无可信参考。F-ICL 提供一个精确的贝叶斯最优参考:我们在图灵完备机器 F 上穷举所有长度不超过13的8600万条有效程序,经补集对称化消除输出极性偏差,并在指定有界 Levin-Solomonoff 先验下计算精确后验。该参考是针对该先验的贝叶斯最优解,而非通用解,且模型从未被告知其存在,因此得分反映模型自身隐含归纳先验。在涵盖0.8B至675B参数的105种开放模型与前沿系统中,模型最高正确率达92%,但45/46个暴露分布均比随机键入参考更远。此非任务选择偏差:在位坐标任务中,半数长度配额无法扭曲,69/80次运行仍低于基准。保真度不随规模提升,而准确率却在上升;续写能改善晚期表现但不收敛;模型平均每两步就破坏一个已解决任务,而F参考每九步才一次,且总能修复。由于绝对距离依赖参考,我们证明了序列界:任何给参考正权重的预测器累积超额损失有界;在不调用参考的损失函数下,任何赋予真实结果正质量的贝叶斯混合都有有界的真值损失预算。在23,998条轨迹中,86.7%已耗超10比特。位置9前结束的序列无法排除任意大的有限常数,故为对手先验必须已支付的下限。F-ICL 是开源基准与工具包。
原文摘要 · Abstract (English)
Whether large language models perform algorithmic inference or pattern completion is hard to test, because most benchmarks supply answers but no distributional reference for what the shown evidence licenses. F-ICL supplies one exactly: we exhaustively enumerate the 86 million valid programs of length at most 13 on a Turing-complete machine F, complement-symmetrised to remove output-polarity bias, and compute the exact posterior under a declared bounded Levin--Solomonoff prior. It is Bayes-optimal for that stated prior rather than universal, and models are never told it exists, so the score reads the inductive prior their served distribution already encodes. Across 105 serving configurations spanning open models from 0.8B to 675B and frontier systems, models answer up to 92% of queries correctly, yet 45 of the 46 exposing distributions sit farther from the F reference than a keystroke reference. This is not an artefact of task selection: on the bit coordinate, the half the length quota cannot distort, 69 of 80 runs stay below the anchor. Fidelity is inert to scale, which accuracy tracks; continuation improves late without converging; and models un-solve a solved task once per two gains, where the F reference does so once per nine and always repairs it. Because absolute distances are reference-dependent, we prove sequential bounds holding for rival priors: any predictor whose prior gives the reference positive weight has bounded cumulative excess loss, and, in a loss never invoking the reference, any Bayesian mixture giving the realised truth positive mass has a bounded truth-loss budget. On 23,998 trajectories, 86.7% already spend over 10 bits of it. Sequences ending by position nine cannot exclude an arbitrarily large finite constant, so these are lower bounds on what a rival prior must already pay. F-ICL is an open benchmark and toolkit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。