arXiv:2607.15277cs.CL2026-07

检验大模型在分层推理中是否满足概率一致性,发现其聚合结果常出错。

Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

论文配图:Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models
图 1 · 摘自论文原文
  • 用二叉树分层划分群体,通过提示词让模型逐层估计子群体分布。
  • 多领域测试显示模型聚合结果普遍违反概率基本定律,误差显著。
  • 细粒度子群体估计反而更准,揭示模型存在‘宏观误判’现象,适合评估模型推理可靠性。

上下文学习常被解释为条件推断:提示词定义上下文,模型输出被视为对应条件分布的估计。若此解释成立,大模型的估计应满足基本概率恒等式。特别是全概率律指出,加权条件分布应聚合为任意有效划分下的总体边缘分布。本文以二叉树为评估框架,递归地将总体划分为更细粒度的子群体,通过自然语言描述子群体并提示模型,再将结果回聚至总体水平,比较不同粒度划分下的估计差异。在多个问题领域及前沿模型上,均发现严重违反一致性原则的现象。深入研究角色提示发现一种称为‘宏观谬误’的模式:从更细粒度子群体回复重建的估计,往往比直接总体估计更接近人类参考数据。该效应在树结构和任务变化下持续存在,可通过隐式提示部分恢复。结果表明,模型虽掌握子群体知识,但无法可靠传递至聚合估计。这一差距确立了统计自洽性作为无参照、未饱和的模型评估标准。

原文摘要 · Abstract (English)

In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Applying this protocol across problem domains and state-of-the-art frontier models, we show widespread violations of basic consistency properties. An in-depth study of persona prompting reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect persists across variations in tree structure and estimation task, and can be partially recovered through implicit prompting. Together, these findings suggest that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

大模型评估概率一致性上下文学习推理验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。