KEDiT让大模型高效接入新知识，对话更准确
Efficient Tuning of Large Language Models for Knowledge-Grounded Dialogue Generation
- 用信息瓶颈压缩检索知识，只保留关键信息
- 仅更新模型2%参数，就能生成高质量对话
- 适合医疗等需实时知识的场景，效果优于现有方法
大型语言模型虽具备强大文本理解与生成能力，但往往无法使用训练数据之外的最新或领域特定知识。为此，我们提出KEDiT，一种针对知识驱动对话生成的高效微调方法。该方法分为两阶段：首先，利用信息瓶颈将检索到的知识压缩为可学习参数，保留关键信息的同时降低计算开销；其次，通过轻量级知识感知适配器，在微调过程中将压缩后的知识向量融入大模型，仅更新少于2%的模型参数。在Wizard of Wikipedia和新构建的PubMed-Dialog数据集上的实验表明，KEDiT在自动评估、基于大模型的评估及人工评估中均显著优于对比基线，能生成上下文相关且信息丰富的回答。该方法有效结合了预训练模型的优势与动态知识融合的灵活性，为医学等领域提供了可扩展的解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate remarkable text comprehension and generation capabilities but often lack the ability to utilize up-to-date or domain-specific knowledge not included in their training data. To address this gap, we introduce KEDiT, an efficient method for fine-tuning LLMs for knowledge-grounded dialogue generation. KEDiT operates in two main phases: first, it employs an information bottleneck to compress retrieved knowledge into learnable parameters, retaining essential information while minimizing computational overhead. Second, a lightweight knowledge-aware adapter integrates these compressed knowledge vectors into the LLM during fine-tuning, updating less than 2\% of the model parameters. The experimental results on the Wizard of Wikipedia and a newly constructed PubMed-Dialog dataset demonstrate that KEDiT excels in generating contextually relevant and informative responses, outperforming competitive baselines in automatic, LLM-based, and human evaluations. This approach effectively combines the strengths of pretrained LLMs with the adaptability needed for incorporating dynamic knowledge, presenting a scalable solution for fields such as medicine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。