arXiv:2506.12367cs.CLcs.SI2025-06被引 4

研究知识图谱提取错误如何影响下游分析,发现错误会系统性扭曲结果。

Understanding the Effect of Knowledge Graph Extraction Error on Downstream Graph Analyses: A Case Study on Affiliation Graphs

  • 从微观边准确率到宏观结构指标,双重评估提取误差影响
  • 错误率上升时,多数分析指标出现稳定方向的高估或低估偏差
  • 现有误差模型无法复现真实偏差模式,需更贴近现实的建模方法

知识图谱(KG)在社会结构、社区动态和机构成员关系等跨领域分析中具有重要价值。尽管大语言模型(LLMs)提升了从大规模文本中自动化抽取知识图谱的可扩展性与可及性,但提取错误对下游分析的影响仍不明确,尤其对依赖精准知识图谱获取现实洞察的应用科学家而言。为填补这一空白,我们首次在两个层面评估了知识图谱提取性能:(1) 微观层面的边准确率,符合标准NLP评估方式,并通过人工识别常见错误来源;(2) 宏观层面的图结构指标,如社区检测和连通性,与实际应用密切相关。以从社会登记簿中提取的人物组织隶属关系图为例,研究发现,在一定提取性能范围内,多数下游图分析指标的偏差接近零。然而,随着提取性能下降,多个指标显示出日益显著的系统性偏差,且每项指标倾向于一致方向的高估或低估。通过模拟实验进一步表明,文献中常用的误差模型无法捕捉这些偏差模式,凸显了构建更真实误差模型的必要性。研究为实践者提供了可操作洞见,强调提升提取方法与误差建模对保障下游分析可靠性与意义的重要性。

原文摘要 · Abstract (English)

Knowledge graphs (KGs) are useful for analyzing social structures, community dynamics, institutional memberships, and other complex relationships across domains from sociology to public health. While recent advances in large language models (LLMs) have improved the scalability and accessibility of automated KG extraction from large text corpora, the impacts of extraction errors on downstream analyses are poorly understood, especially for applied scientists who depend on accurate KGs for real-world insights. To address this gap, we conducted the first evaluation of KG extraction performance at two levels: (1) micro-level edge accuracy, which is consistent with standard NLP evaluations, and manual identification of common error sources; (2) macro-level graph metrics that assess structural properties such as community detection and connectivity, which are relevant to real-world applications. Focusing on affiliation graphs of person membership in organizations extracted from social register books, our study identifies a range of extraction performance where biases across most downstream graph analysis metrics are near zero. However, as extraction performance declines, we find that many metrics exhibit increasingly pronounced biases, with each metric tending toward a consistent direction of either over- or under-estimation. Through simulations, we further show that error models commonly used in the literature do not capture these bias patterns, indicating the need for more realistic error models for KG extraction. Our findings provide actionable insights for practitioners and underscores the importance of advancing extraction methods and error modeling to ensure reliable and meaningful downstream analyses.

知识图谱误差分析图分析大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。