用紧凑离散符号替代大模型参数中的知识，提升可读性和效率
KoRe: Compact Knowledge Representations for Large Language Models
- 将一跳知识图谱转为离散符号注入大模型
- 在三个基准上表现优异，令牌使用量最多减少10倍
- 适合需要可解释、易更新知识的大模型应用
现代大型语言模型在问答等用户任务中表现出色，推理能力持续提升。然而，其知识存储方式存在根本缺陷：世界知识被编码在模型参数中，导致表达不透明、难以调试和更新，且易产生幻觉。相比之下，知识图谱提供可读性强、易于编辑的知识表示，应用于知识密集型任务时能显著提升下游性能。但现有融合方法需大量重训练或微调。为此，我们提出KoRe，一种将一跳子图编码为紧凑离散知识令牌并注入大模型骨干的方法。在三个主流基准上测试，结果表明该方法性能竞争性强，同时令牌使用量最多减少10倍。结果证明，紧凑离散的知识图谱表示可高效有效地为现代大模型提供知识支撑。
原文摘要 · Abstract (English)
Modern Large Language Models (LLMs) have shown impressive performances in user-facing tasks such as question answering, as well as consistent improvements in reasoning capabilities. Still, the way these models encode knowledge seems inherently flawed: by design, LLMs encode world-knowledge within their parameters. This way of representing knowledge is inherently opaque, difficult to debug and update, and prone to hallucinations. On the other hand, Knowledge Graphs can provide human-readable and easily editable world knowledge representations, and their application in knowledge-intensive tasks has consistently proven beneficial to downstream performance. Nonetheless, current integration techniques require extensive retraining or finetuning. To overcome this issue, we introduce KoRe, a methodology to encode 1-hop sub-graphs into compact discrete knowledge tokens and inject them into a LLM backbone. We test the proposed approach on three established benchmarks, and report competitive performances coupled with a significant reduction (up to 10x) in token usage. Our results show that compact discrete KG representations can efficiently and effectively be used to ground modern LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。