用RAG原理微调小模型，实现记忆增强的智能问答
Tuning LLMs by RAG Principles: Towards LLM-native Memory
- 用RAG生成数据微调7B小模型，融合长上下文与检索优势
- 在三组数据上超越大模型和传统RAG方法，覆盖多种查询类型
- 适合需要精准记忆且资源受限的部署场景
大语言模型（LLM）的额外记忆对个人助理等实际应用至关重要。当前主流方案为长上下文模型与检索增强生成（RAG）。本文在三个新构建的数据集上系统比较二者：（1）长上下文模型虽成本高，但更擅长整体把握记忆信息，适合需全局理解的查询；（2）当查询聚焦具体信息时，若关键词可明确匹配，RAG更具优势。为此，本文提出RAG-Tuned-LLM方法，使用遵循RAG原则生成的数据微调小型（如7B）LLM，使其兼具两者优点。在三个数据集上的大量实验表明，RAG-Tuned-LLM在各类查询中均优于长上下文模型和传统RAG方法。
原文摘要 · Abstract (English)
Memory, additional information beyond the training of large language models (LLMs), is crucial to various real-world applications, such as personal assistant. The two mainstream solutions to incorporate memory into the generation process are long-context LLMs and retrieval-augmented generation (RAG). In this paper, we first systematically compare these two types of solutions on three renovated/new datasets and show that (1) long-context solutions, although more expensive, shall be easier to capture the big picture and better answer queries which require considering the memory as a whole; and (2) when the queries concern specific information, RAG solutions shall be more competitive especially when the keywords can be explicitly matched. Therefore, we propose a novel method RAG-Tuned-LLM which fine-tunes a relative small (e.g., 7B) LLM using the data generated following the RAG principles, so it can combine the advantages of both solutions. Extensive experiments on three datasets demonstrate that RAG-Tuned-LLM can beat long-context LLMs and RAG methods across a wide range of query types.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。