区分知识缺失与记忆失效，发现大模型主要卡在回忆环节。
Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality
- 按事实级分析模型知识：是否存储、能否调用、需推理才能回忆。
- 前沿模型编码率达95%-98%，但回忆失败仍普遍存在。
- 推理能显著提升召回率，适合改进模型利用已有知识的方法。
主流大模型事实性评估将所有错误一视同仁，掩盖了错误源于知识缺失（空架子）还是访问受限（钥匙丢失）。本文提出一种行为框架，以事实为单位刻画知识状态：是否编码，以及可访问性——无法回忆、直接回忆或需推理才能回忆。为此构建了自动化生成的基准WikiProfile，基于网络搜索引导的提示语言模型生成。在13个大模型对400万条回答的测试中发现，前沿模型在该基准上编码率接近饱和，GPT-5与Gemini-3达95%–98%。然而，回忆仍是主要瓶颈：许多曾归因于知识缺失的错误实为访问失败。此类失败具有系统性，尤其影响长尾事实和反向问题。最后，我们证明推理可显著提升回忆能力，恢复大量失败案例，表明未来性能提升可能更依赖优化知识利用，而非单纯扩展规模。
原文摘要 · Abstract (English)
Standard factuality evaluations of LLMs treat all errors alike, obscuring whether failures arise from missing knowledge (empty shelves) or from limited access to encoded facts (lost keys). We propose a behavioral framework that profiles factual knowledge at the level of facts rather than questions, characterizing each fact by whether it is encoded, and then by how accessible it is: cannot be recalled, can be directly recalled, or can only be recalled with inference-time computation (thinking). To support such profiling, we introduce WikiProfile, a new benchmark constructed via an automated pipeline with a prompted LLM grounded in web search. Across 4 million responses from 13 LLMs, we find that encoding is nearly saturated in frontier models on our benchmark, with GPT-5 and Gemini-3 encoding 95--98% of facts. However, recall remains a major bottleneck: many errors previously attributed to missing knowledge instead stem from failures to access it. These failures are systematic and disproportionately affect long-tail facts and reverse questions. Finally, we show that thinking improves recall and can recover a substantial fraction of failures, indicating that future gains may rely less on scaling and more on methods that improve how models utilize what they already encode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。