arXiv:2603.08274cs.CLcs.AI2026-03被引 3

1720亿token实测发现:大模型在文档问答中幻觉率随上下文增长而飙升,模型选型比温度设置更关键。

How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

  • 用真实答案先行法评估,避免人工标注偏差
  • 200K上下文时幻觉率超10%,是32K时的近三倍
  • 模型选型影响最大,硬件平台无显著差异

大语言模型在基于文档的问答中实际会产生多少幻觉?尽管这对企业AI部署至关重要,但可靠测量因依赖易污染的静态数据集、有偏见的LLM评分器或样本量不足而受阻。本文采用RIKER方法,实现无需人工标注的确定性评分。在35个开源模型、三种上下文长度(32K、128K、200K token)、四种温度设置及三种硬件平台(NVIDIA H200、AMD MI300X、Intel Gaudi 3)上,完成了超过1720亿token的评估,规模远超以往。结果表明:(1)即使表现最佳的模型在32K上下文中仍存在1.19%的幻觉率,顶级模型达5%-7%;上下文增至128K时幻觉率几乎翻倍,200K时所有模型均超过10%;(2)模型选择是决定因素,整体准确率跨度达72个百分点,模型家族比模型大小更能预测抗幻觉能力;(3)温度影响复杂:T=0.0在约60%情况下准确率最高,但高温度可降低幻觉率并显著减少无限生成循环(最高达T=0.0的48倍);(4)事实检索与抗幻觉是独立能力,能找对事实的模型仍可能虚构不存在的信息;(5)结果在不同硬件平台间一致,部署无需考虑硬件差异。

原文摘要 · Abstract (English)

How much do large language models actually hallucinate when answering questions grounded in provided documents? Despite the critical importance of this question for enterprise AI deployments, reliable measurement has been hampered by benchmarks that rely on static datasets vulnerable to contamination, LLM-based judges with documented biases, or evaluation scales too small for statistical confidence. We address this gap using RIKER, a ground-truth-first evaluation methodology that enables deterministic scoring without human annotation. Across 35 open-weight models, three context lengths (32K, 128K, and 200K tokens), four temperature settings, and three hardware platforms (NVIDIA H200, AMD MI300X, and Intel Gaudi 3), we conducted over 172 billion tokens of evaluation - an order of magnitude beyond prior work. Our findings reveal that: (1) even the best-performing models fabricate answers at a non-trivial rate - 1.19% at best at 32K, with top-tier models at 5 - 7% - and fabrication rises steeply with context length, nearly tripling at 128K and exceeding 10% for all models at 200K; (2) model selection dominates all other factors, with overall accuracy spanning a 72-percentage-point range and model family predicting fabrication resistance better than model size; (3) temperature effects are nuanced - T=0.0 yields the best overall accuracy in roughly 60% of cases, but higher temperatures reduce fabrication for the majority of models and dramatically reduce coherence loss (infinite generation loops), which can reach 48x higher rates at T=0.0 versus T=1.0; (4) grounding ability and fabrication resistance are distinct capabilities - models that excel at finding facts may still fabricate facts that do not exist; and (5) results are consistent across hardware platforms, confirming that deployment decisions need not be hardware-dependent.

幻觉检测大模型评测文档问答推理质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。