arXiv:2412.18443cs.CL2024-12被引 2

大模型在三元组集合预测中易产生幻觉,导致性能下降。

Is Large Language Model Good at Triple Set Prediction? An Empirical Study

  • 用大模型挖掘知识图谱规则,再基于子图推理缺失三元组
  • 在CFamily数据集上,大模型预测时幻觉严重,性能明显下降
  • 适合研究大模型在知识推理中的可靠性与局限性

知识图谱补全(KGC)的核心是预测缺失的关系或节点。传统KGC任务多基于已知一两个元素推断未知元素,而三元组集合预测(TSP)更贴近真实场景,旨在仅凭已知三元组信息预测所有缺失元素。近年来大语言模型(LLM)在语言理解上取得显著进展,展现出解决KGC的潜力,但其在TSP任务中的表现尚未被系统研究。本文提出一种新框架,包含基于LLM的规则挖掘和基于规则的三元组集合预测。将富含语义的信息关系列表用于提示LLM生成规则,该过程高效且不依赖统计信息,可挖掘有效、真实的规则。对每个子图,结合相关三元组与指定规则,引导LLM预测缺失三元组,最终整合各子图结果得到完整预测集。在较完整的CFamily数据集上评估发现,当要求LLM依据大量事实知识预测时,出现显著幻觉,导致性能大幅下降。通过详尽案例分析,进一步探讨了该现象成因。

原文摘要 · Abstract (English)

The core of the Knowledge Graph Completion (KGC) task is to predict and complete the missing relations or nodes in a KG. Common KGC tasks are mostly about inferring unknown elements with one or two elements being known in a triple. In comparison, the Triple Set Prediction (TSP) task is a more realistic knowledge graph completion task. It aims to predict all elements of unknown triples based on the information from known triples. In recent years, large language models (LLMs) have exhibited significant advancements in language comprehension, demonstrating considerable potential for KGC tasks. However, the potential of LLM on the TSP task has not yet to be investigated. Thus in this paper we proposed a new framework to explore the strengths and limitations of LLM in the TSP task. Specifically, the framework consists of LLM-based rule mining and LLM-based triple set prediction. The relation list of KG embedded within rich semantic information is first leveraged to prompt LLM in the generation of rules. This process is both efficient and independent of statistical information, making it easier to mine effective and realistic rules. For each subgraph, the specified rule is applied in conjunction with the relevant triples within that subgraph to guide the LLM in predicting the missing triples. Subsequently, the predictions from all subgraphs are consolidated to derive the complete set of predicted triples on KG. Finally, the method is evaluated on the relatively complete CFamily dataset. The experimental results indicate that when LLMs are required to adhere to a large amount of factual knowledge to predict missing triples, significant hallucinations occurs, leading to a noticeable decline in performance. To further explore the causes of this phenomenon, this paper presents a comprehensive analysis supported by a detailed case study.

知识图谱大模型三元组预测幻觉

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。