用知识库增强大模型，无需检索模块且可动态更新。
KBLaM: Knowledge Base augmented Language Model
- 将知识库转为向量对，通过特殊注意力机制注入大模型
- 支持10,000+三元组知识注入80亿参数模型，仅需单张A100 GPU
- 适合需要实时更新知识的大模型应用，如问答与推理任务
本文提出知识库增强语言模型（KBLaM），一种将外部知识融入大语言模型的新方法。KBLaM基于文档语料构建知识库，利用预训练句向量编码器与线性适配器将知识库中每条知识转换为连续的键值向量对，并通过专用的矩形注意力机制集成到预训练语言模型中。与检索增强生成不同，KBLaM无需外部检索模块；与上下文学习相比，其计算开销随知识库规模线性增长而非平方级增长。该方法可在单张A100 80GB GPU上，将超过10,000个三元组的知识库注入一个80亿参数、上下文窗口仅8,000的预训练模型，并支持无需微调或重训练的动态更新。实验表明，KBLaM在问答和开放式推理等任务中表现优异，且能提供对知识使用过程的可解释性洞察。代码与数据集已公开于https://github.com/microsoft/KBLaM/
原文摘要 · Abstract (English)
In this paper, we propose Knowledge Base augmented Language Model (KBLaM), a new method for augmenting Large Language Models (LLMs) with external knowledge. KBLaM works with a knowledge base (KB) constructed from a corpus of documents, transforming each piece of knowledge in the KB into continuous key-value vector pairs via pre-trained sentence encoders with linear adapters and integrating them into pre-trained LLMs via a specialized rectangular attention mechanism. Unlike Retrieval-Augmented Generation, KBLaM eliminates external retrieval modules, and unlike in-context learning, its computational overhead scales linearly with KB size rather than quadratically. Our approach enables integrating a large KB of more than 10K triples into an 8B pre-trained LLM of only 8K context window on one single A100 80GB GPU and allows for dynamic updates without model fine-tuning or retraining. Experiments demonstrate KBLaM's effectiveness in various tasks, including question-answering and open-ended reasoning, while providing interpretable insights into its use of the augmented knowledge. Code and datasets are available at https://github.com/microsoft/KBLaM/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。