用新方法系统评估大模型修复知识图谱的准确性。
Systematic Evaluation of Knowledge Graph Repair with Large Language Models
- 设计违规操作生成器，自动构造符合SHACL约束的错误数据
- 大模型修复系统在简洁提示下表现最佳，需包含约束与上下文
- 适合研究知识图谱修复与大模型应用的开发者
我们提出一种系统化方法，用于评估知识图谱修复质量，依据形状约束语言（SHACL）定义的约束违规。现有评估依赖临时构建的数据集，难以在更广泛场景中进行严谨分析。本方法通过一种新型机制——违规诱导操作（VIOs），系统生成各类违规。利用该框架，我们评估了基于大语言模型构建的多种修复系统，并分析不同提示策略的表现。结果表明，同时包含相关违反的SHACL约束和知识图谱关键上下文信息的简洁提示，能带来最优修复效果。
原文摘要 · Abstract (English)
We present a systematic approach for evaluating the quality of knowledge graph repairs with respect to constraint violations defined in shapes constraint language (SHACL). Current evaluation methods rely on \emph{ad hoc} datasets, which limits the rigorous analysis of repair systems in more general settings. Our method addresses this gap by systematically generating violations using a novel mechanism, termed violation-inducing operations (VIOs). We use the proposed evaluation framework to assess a range of repair systems which we build using large language models. We analyze the performance of these systems across different prompting strategies. Results indicate that concise prompts containing both the relevant violated SHACL constraints and key contextual information from the knowledge graph yield the best performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。