用知识图谱嵌入减少大模型幻觉，不需微调主模型
Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality
- 将输入文本转为知识图谱嵌入，通过适配器融入语言模型
- 在HaluEval、FEVER等基准上提升事实准确性，幻觉率下降
- 无需外部检索或微调主模型，适合快速部署到各类LLM
本文提出一种通过引入知识图谱（KG）作为额外模态来降低大语言模型（LLM）幻觉的方法。该方法将输入文本转换为知识图谱嵌入，并使用适配器将其整合到语言模型空间中，无需依赖外部检索。为此，我们构建了WikiEntities数据集，包含超过300万条维基百科文本，标注了维基数据中的实体及其来自PyTorch-BigGraph的嵌入。该数据集可用于训练实体链接模型，并适配至多种LLM。我们的方法无需微调语言模型本身，仅训练适配器，避免影响原模型在其他任务上的表现。我们在Mistral 7B、LLaMA 2-7B（聊天版）和LLaMA 3-8B（指令版）上训练了适配器，结果表明该方法在HaluEval、True-False和FEVER数据集上均显著提升事实准确性，证明将知识图谱作为新模态能有效减少幻觉，且无需外部检索。
原文摘要 · Abstract (English)
In this paper we present an approach to reduce hallucinations in Large Language Models (LLMs) by incorporating Knowledge Graphs (KGs) as an additional modality. Our method involves transforming input text into a set of KG embeddings and using an adapter to integrate these embeddings into the language model space, without relying on external retrieval processes. To facilitate this, we created WikiEntities, a dataset containing over 3 million Wikipedia texts annotated with entities from Wikidata and their corresponding embeddings from PyTorch-BigGraph. This dataset serves as a valuable resource for training Entity Linking models and adapting the described method to various LLMs using specialized adapters. Our method does not require fine-tuning of the language models themselves; instead, we only train the adapter. This ensures that the model's performance on other tasks is not affected. We trained an adapter for the Mistral 7B, LLaMA 2-7B (chat), and LLaMA 3-8B (instruct) models using this dataset and demonstrated that our approach improves performance on the HaluEval, True-False benchmarks and FEVER dataset. The results indicate that incorporating KGs as a new modality can effectively reduce hallucinations and improve the factual accuracy of language models, all without the need for external retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。