测试大模型在证据不全时能否正确判断'不存在',发现多数模型会错误认定缺失信息。
When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Models
- 设计新数据集CROWN-QA,控制覆盖范围差异以测试模型推理能力
- 三类模型普遍存在过度判断,约70%在部分覆盖时仍给出否定答案
- 提示工程无法根治错误,结构化验证可定位证据覆盖误判根源
大型语言模型常被问及某事物是否不在记录、列表或检索上下文中。然而,仅当证据完整覆盖查询范围时,否定回答才合理;否则应为未知。我们称之为完整性敏感的否定推理。本文提出CROWN-QA,包含两个部分:CROWN-Synth(控制变量的合成数据对,仅改变查询相关覆盖率)和CROWN-Real(真实文档上的对比评估,具有受控覆盖率变体)。在三个主流LLM家族中,模型表现出不稳定的闭合判断,存在显著的过度闭合现象,无法可靠区分合理否定(认证否定)与证据不足(未知)。主要失败模式呈非对称性:模型常识别隐含完整证据,却将隐含不完整的证据误认为覆盖查询。提示工程仅重新分配过闭合与欠闭合错误,未根本解决。结构化证书提取显示多数错误源于对证据覆盖范围的误判。CROWN-Real表明,核心的不完整覆盖不对称性在真实文档中依然存在,其强度及过/欠闭合平衡随模型、提示和来源而异。
原文摘要 · Abstract (English)
Large language models (LLMs) are often asked whether something is absent from a record, list, or retrieved context. Yet non-observation licenses a negative answer only when evidence completely covers the query scope; otherwise, the answer should remain unknown. We call this completeness-sensitive negative reasoning. We introduce CROWN-QA, comprising CROWN-Synth, a controlled paired core that fixes the question and observed facts while varying only query-relative coverage, and CROWN-Real, a real-document contrast-set evaluation with controlled coverage variants. Across three LLM families, models show unstable closure judgments and substantial over-closure, failing to reliably distinguish a justified negative answer (Certified-Negative) from insufficient evidence (Unknown). The dominant CROWN-Synth failure is asymmetric: models often recognize implicitly complete evidence yet treat implicitly partial evidence as query-covering. Prompting redistributes errors between over- and under-closure rather than consistently resolving them. Structured certificate elicitation traces many errors to evidence-coverage mischaracterization. CROWN-Real shows that the core partial-coverage asymmetry persists on real-document content, while its strength and the balance between over- and under-closure vary by model, prompt, and source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。