arXiv:2607.10212cs.AIcs.CL2026-07

提出可解释的评估框架,精准检测知识图谱构建中的遗漏与冗余。

KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text

论文配图:KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text
图 1 · 摘自论文原文
  • 基于实体与关系双层面评估,结合语法对齐与否定处理机制。
  • 在多个数据集上识别出传统指标忽略的结构偏差与语义失真。
  • 适合研究知识图谱抽取方法或需标准化评估的研究者使用。

知识图谱常通过自动化抽取流程构建,但系统易引入虚假或不完整的三元组,影响下游性能。现有评估依赖任务特定指标或小规模人工验证,难以揭示图谱的结构与语义保真度。本文提出一种新型、可解释的内在质量评估指标,衡量自动抽取图谱与源文本中关键名词短语、谓词关系及否定等语言现象所构成的理想图谱之间的逼近程度。框架包含两个互补组件:(1) 实体级评估,涵盖完整性、消歧质量与连通性;(2) 关系级评估,通过词汇相似性、依存句法对齐与轻量否定处理,判断谓词保留性与多重性。我们在 WebNLG、TinyButMighty 与 BenchIE 等多个先进三元组抽取系统与数据集上验证,结果表明该指标能可靠发现传统方法遗漏的缺失、冗余与结构偏差。进一步通过消融实验与下游链接预测任务验证,显示 KGCQual 分数与性能显著相关。代码已开源。

原文摘要 · Abstract (English)

Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs. We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an "ideal" graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text. Our framework integrates two complementary components: (1) an entity-level assessment that evaluates completeness, resolution quality, and connectivity, and (2) a relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and light-weight negation handling to ensure semantic faithfulness. We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook. Our work offers a scalable, model-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardised evaluation. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance on the same extracted KGs. The code repository is available at https://github.com/kracr/kg-quality-metric.

知识图谱质量评估可解释性自然语言理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。