用外部文档验证知识图谱三元组，让大模型推理过程可追溯。
Traceable LLM-based validation of statements in knowledge graphs
- 通过检索外部文档替代大模型内部知识进行三元组验证
- 在BioRED数据集上达到88%精确率，44%召回率
- 适合需要可解释性的知识图谱自动化校验场景
本文提出一种基于大模型的RDF三元组验证方法，强调推理过程的可追溯性。由于大模型无法可靠追踪信息来源，本方法不依赖其内部知识，而是将待验证的三元组与通过网络搜索或维基百科检索到的文本片段进行比对。为评估该检索增强生成（RAG）流程在生物科学内容上的适用性，我们在BioRED数据集上测试了1,719条正向陈述和同等数量的负向新生成陈述，结果精确率为88%，召回率为44%。此外，在SNLI数据集上对比了专用于自然语言推断任务的模型，验证了本方法的有效性。实验还展示了在Wikidata上的应用，利用SPARQL查询自动提取待验证陈述。结果表明，该方法可实现知识图谱的大规模自动化验证，显著降低人工标注成本。
原文摘要 · Abstract (English)
This article presents a method for verifying RDF triples using LLMs, with an emphasis on providing traceable arguments. Because the LLMs cannot currently reliably identify the origin of the information used to construct the response to the user prompt, our approach is to avoid using internal LLM factual knowledge altogether. Instead, verified RDF statements are compared to chunks of external documents retrieved through a web search or Wikipedia. To assess the possible application of this retrieval augmented generation (RAG) workflow on biosciences content, we evaluated 1,719 positive statements from the BioRED dataset and the same number of newly generated negative statements. The resulting precision is 88 %, and recall is 44 %. This indicates that the method requires human oversight. We also evaluated the method on the SNLI dataset, which allowed us to compare our approach with models specifically tuned for the natural language inference task. We demonstrate the method on Wikidata, where a SPARQL query is used to automatically retrieve statements needing verification. Overall, the results suggest that LLMs could be used for large-scale verification of statements in KGs, a task previously unfeasible due to human annotation costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。