首次全面构建大模型知识图谱，揭示其知识范围与结构
Enabling LLM Knowledge Analysis via Extensive Materialization
- 通过递归查询与结果整合，系统化提取模型内部知识
- 建成包含1.01亿条三元组的知识库，覆盖290万实体
- 适合研究大模型知识特性、偏见与一致性的人士
大语言模型（LLMs）在自然语言处理与人工智能领域取得重大进展，其内化事实知识的能力是关键成功因素之一。自Petroni等人（2019）以来，对模型知识的分析受到关注，但多数方法仅针对单个问题进行小规模预设样本测试，导致‘可得性偏差’，限制了对模型知识（或信念）的全面考察。为此，本文提出一种新方法，通过递归查询与结果合并，全面物化大模型的事实知识。该方法是大模型研究的重要里程碑，首次提供了关于模型知识范围与结构的实质性洞见。作为原型，我们构建了GPTKB，一个基于GPT-4o-mini的知识库，包含1.01亿条关系三元组，覆盖超过290万实体。利用GPTKB，我们同时分析了GPT-4o-mini在知识规模、准确性、偏见、截止点和一致性方面的表现。GPTKB已公开：https://gptkb.org
原文摘要 · Abstract (English)
Large language models (LLMs) have majorly advanced NLP and AI, and next to their ability to perform a wide range of procedural tasks, a major success factor is their internalized factual knowledge. Since Petroni et al. (2019), analyzing this knowledge has gained attention. However, most approaches investigate one question at a time via modest-sized pre-defined samples, introducing an ``availability bias'' (Tversky&Kahnemann, 1973) that prevents the analysis of knowledge (or beliefs) of LLMs beyond the experimenter's predisposition. To address this challenge, we propose a novel methodology to comprehensively materialize an LLM's factual knowledge through recursive querying and result consolidation. Our approach is a milestone for LLM research, for the first time providing constructive insights into the scope and structure of LLM knowledge (or beliefs). As a prototype, we build GPTKB, a knowledge base (KB) comprising 101 million relational triples for over 2.9 million entities from GPT-4o-mini. We use GPTKB to exemplarily analyze GPT-4o-mini's factual knowledge in terms of scale, accuracy, bias, cutoff and consistency, at the same time. GPTKB is accessible at https://gptkb.org
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。