用真实证据验证大模型知识，发现其准确率远低于测试基准。
LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale
- 从模型参数中提取130万条百科内容，逐条比对维基和网络证据。
- gpt-5-mini在维基覆盖主题上仅68.4%可验证为真，比MMLU低21个百分点。
- 揭示模型知识存在大量无法验证的空白，适合关注可信AI的研究者。
MMLU等基准显示顶级语言模型的事实性已超过90%。但《LLMpedia》表明这一图景并不完整。我们从三个模型家族的参数记忆中完全提取约130万条百科文章，并逐条与维基百科及精心筛选的网络证据进行审计。对于gpt-5-mini，在维基百科覆盖的主题上，可验证的真实率为68.4%，较MMLU低超过21个百分点，且差距主要源于不可验证性（30.5%），而非错误陈述（1.2%）。在更前沿的文章上，与精选网络证据对比时，准确率可达57.6%；而维基百科仅覆盖模型生成主题的56.7%，三类模型主题重合度仅为7.3%。在受先前对Grokipedia分析启发的检索陷阱测试中，LLMpedia在文本相似度仅为维基一半时仍更准确。所有提示、文章与判断结果均已公开。数据、代码与界面：https://llmpedia.net。
原文摘要 · Abstract (English)
Benchmarks like MMLU suggest flagship language models approach factuality saturation above 90\%. \emph{LLMpedia} shows this picture is incomplete. We materialize ${\sim}$1.3M encyclopedia articles entirely from parametric memory across three model families, then audit every claim against Wikipedia and curated web evidence. For \texttt{gpt-5-mini}, the verifiable true rate is 68.4\% on Wikipedia-covered subjects - more than 21\,pp below MMLU - and the gap is driven by \emph{unverifiability} (30.5\%), not refutation (1.2\%). Beyond Wikipedia, frontier articles audited against curated web evidence reach 57.6\%; Wikipedia covers only 56.7\% of model-surfaced subjects, and three model families overlap in just 7.3\% of subject choices. In a retrieval-trap benchmark inspired by prior analysis of Grokipedia, LLMpedia is more factual at roughly half the textual similarity to Wikipedia. Every prompt, article, and verdict is released. Data, code, interface: https://llmpedia.net.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。