arXiv:2606.07537cs.CLcs.AI2026-06

揭示大模型幻觉的三大结构根源及数据放大效应

From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data

论文配图:From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data
图 1 · 摘自论文原文
  • 自注意力机制导致实体混淆与语义漂移
  • 最大似然训练奖励统计可能而非事实正确
  • 自回归解码使错误逐词传播且无法修正

大型语言模型会产生流畅且自信的虚假输出,这种幻觉现象在不同生成阶段和模型规模间具有持续性。现有分类体系按输出类型区分内在与外在错误,但无法定位具体内部机制。本文分析幻觉是三个架构决策共同作用的结果:自注意力通过共现学习替代语义理解,引发实体混淆、事实误归因与语义漂移;最大似然估计目标优化下一个词的概率,不考虑事实真伪,奖励统计上合理的输出;自回归解码在暴露偏差下强制从左到右单向推进,一个错误词将贯穿整个序列无法修正。数据集缺陷——长尾缺失、训练偏差与合成污染——会放大这些漏洞,但并非独立成因。本文贡献包括:第一,将每种机制对应至Alansari和Luqman分类中的特定输出类型,分别定位内在幻觉于自注意力、外在幻觉于MLE、逻辑不一致于自回归解码;第二,证明常见数据问题均利用了上述机制而非独立引发幻觉;第三,指出现有仅按输出类型分类的局限性,并对比基于推理层的缓解方法。

原文摘要 · Abstract (English)

Large language models hallucinate--producing fluent, confident, factually wrong outputs--with a consistency that persists across generations and scales. Existing taxonomies classify hallucination by output type, distinguishing intrinsic from extrinsic failures and faithfulness from factuality divergence. These frameworks are descriptively rigorous but do not identify which internal mechanism produced a given instance. This paper analyses hallucination as a structural consequence of three architectural decisions that together form a compound failure system. Self-attention's co-occurrence learning substitutes statistical proximity for semantic meaning and produces entity confusion, fact misattribution, and semantic drift. The maximum likelihood estimation training objective optimises next-token probability without factual constraint, rewarding statistically plausible outputs regardless of their truth value. Autoregressive decoding's permanent left-to-right commitment under exposure bias ensures that a single wrong token cascades forward through the entire output sequence without revision. Dataset pathologies--long-tail deficiencies, training bias, and synthetic pollution--amplify these vulnerabilities but do not independently cause them. We make three contributions. First, we map each mechanism to a specific output category in the Alansari and Luqman taxonomy, locating intrinsic hallucination in self-attention, extrinsic hallucination in MLE, and logical inconsistency in autoregressive decoding. Second, we show that each commonly cited dataset pathology exploits one of these mechanisms rather than originating hallucination independently. Third, we identify the diagnostic limitation of output-type-only classification and contrast it with inference-layer mitigation approaches.

大模型幻觉自注意力训练目标

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。