测试大模型对长尾事实的多版本答案记忆能力,发现多数模型只记住一种说法。
Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge

- 构建可审计的图谱管道,从低曝光网络语料生成含分歧的问答对。
- 32个模型中最强者仅52.4%问题能记住双版本答案,多数只记一个。
- 适合关注模型知识完整性与认知偏见的研究者使用。
事实型问答通常假设单一标准答案,掩盖了大语言模型对长尾事实是否存在多种观点的记忆能力。为此,我们提出ElephantBench,一个闭卷知识探测基准,包含1,094个通过可审计图谱管道生成的问题。该管道从低曝光网络语料中检索相关文档,识别自然发生的观点分歧,并将其转化为多版本问答记录。每个答案均经原始文档和权威公开网页验证,并由人工标注员审核。在32个模型中,即使最强模型也仅在52.4%的问题上同时回忆起两个版本答案,其余多数问题仅能回忆其中一个。模型规模扩大和推理时思维链改进虽提升召回率,但无法消除这种不完整性。语料分析显示,主导观点因曝光更多而占优,少数派观点暴露度越高,模型回忆越完整。研究确立ElephantBench为诊断参数化记忆中认知局限性的可复现工具。更广泛地,我们的图谱构建流程为将长尾语料高效转化为可溯源的知识探测器提供方法支持,助力下一代大模型的认知严谨性评估与提升。代码已开源:https://github.com/Tencent/ElephantBench。
原文摘要 · Abstract (English)
Factual question answering (QA) typically assumes a single canonical answer, obscuring whether large language models (LLMs) retain divergent accounts of long-tail facts. To address this gap, we introduce ElephantBench, a closed-book knowledge probe comprising 1,094 questions generated through an auditable graph-based pipeline. The pipeline retrieves related documents from a low-exposure web corpus, identifies naturally occurring disagreements, and converts them into multi-account QA records. Each answer is verified against the originating documents and authoritative public web sources and is then reviewed by human annotators. Across 32 models, even the strongest model recovers both accounts on only 52.4% of questions, while on nearly all remaining questions it recalls one account but omits the other. Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness. Corpus analysis further shows that exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall. These findings establish ElephantBench as a reproducible knowledge probe for diagnosing epistemic myopia in parametric memory. More broadly, our graph-based benchmark construction pipeline provides an efficient and scalable way to turn long-tail corpora into source-traceable knowledge probes, supporting efforts to evaluate and advance the epistemic rigour of next-generation LLMs. Code is available at https://github.com/Tencent/ElephantBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。