分析1亿条大模型信念，发现其知识存在大量错误与矛盾。
Mining the Mind: What 100M Beliefs Reveal About Frontier LLM Knowledge
- 用1亿条递归提取的信念数据，系统评估大模型事实知识
- 模型准确率远低于以往评测结果，错误率显著
- 揭示知识不一致、模糊和幻觉问题，适合关注模型可信度的研究者
大语言模型在自然语言处理和人工智能任务中表现卓越，其事实知识是关键因素，但目前仍缺乏深入理解，且多数研究基于有偏样本。本文基于GPTKB v1.5（Hu et al., 2025a）——一个从当前最强前沿模型GPT-4.1递归提取的1亿条信念数据集，深入探究该模型的事实知识。研究发现，模型的事实知识与已知知识库存在显著差异，其准确性远低于以往基准测试所显示的水平。此外，不一致性、模糊性和幻觉现象普遍存在，为未来关于大模型事实知识的研究提供了重要方向。
原文摘要 · Abstract (English)
LLMs are remarkable artifacts that have revolutionized a range of NLP and AI tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we take a deep tour into the factual knowledge (or beliefs) of a frontier LLM, based on GPTKB v1.5 (Hu et al., 2025a), a recursively elicited set of 100 million beliefs of one of the strongest currently available frontier LLMs, GPT-4.1. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities concerning factual LLM knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。