用领域知识微调模型，能显著提升对话系统事实准确性。
Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems
- 在RAG和代理系统中引入领域知识微调（KAFT）
- 相比提示工程，事实准确率大幅提升，尤其在复杂场景下
- 首个实证研究该方法的论文，适合做对话系统优化的研究者
大型语言模型（LLMs）已应用于对话系统，但在知识密集型场景下仍易出错。近年来，基于检索增强生成（RAG）和智能体的方案通过外部知识库（KB）增强模型，提升事实准确性，主要依赖提示词、示例和检索到的知识。然而，由于缺乏特定领域的训练，模型难以有效利用这些知识。为此，我们提出知识增强微调（KAFT），在基于RAG和代理的系统中，使用领域数据与外部知识对模型进行微调。我们在真实客户客服数据集MobileCS2上系统比较了提示与KAFT的效果。实验表明，KAFT在两种系统中均显著优于提示，尤其在事实准确性方面。据我们所知，这是首个关于KAFT的扎实实证研究。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently been applied to dialog systems. Despite making progress, LLMs are prone to errors in knowledge-intensive scenarios. Recently, approaches based on retrieval augmented generation (RAG) and agent have emerged to improve the factual accuracy by enhancing the LLMs with knowledge retrieved from external knowledge bases (KBs). This is mostly implemented by prompting the LLMs with instructions, examples and the retrieved knowledge. However, LLMs may have difficulty using the retrieved knowledge effectively for response generation, because they are not well trained to do such generation for specific domains. To mitigate this problem, we propose to finetune the LLMs in the RAG-based and agent-based systems with domain-specific data, together with domain-specific external knowledge, which is called knowledge augmented finetuning (KAFT). We base our study on the MobileCS2 dataset, a real-life customer service dialog dataset that features intensive knowledge interactions, to systematically compare the prompting and KAFT techniques in the RAG-based and agent-based systems. Experiment results show that KAFT substantially surpasses prompting in both RAG and agent systems, particularly in terms of factual accuracy. To the best of our knowledge, this paper represents the first solid empirical work to investigate the KAFT idea.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。