用自生成问题提升大模型在专业领域问答表现
SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains
- 让模型自己生成领域问题并筛选优质样本用于训练
- 在11个数据集上性能提升1.2%至8.6%
- 适合缺乏标注数据的专业领域适配场景
检索增强生成（RAG）通过引入外部知识提升大语言模型（LLM）的问答能力。然而，将通用RAG系统适配至科学、医学等专业领域面临分布偏移和领域数据稀缺的挑战。为此，我们提出SimRAG，一种自训练方法，使LLM同时具备问答与自动生成问题的能力以实现领域适应。首先，在指令遵循、问答和搜索相关数据上微调LLM；随后，利用同一模型从无标签语料中生成多样化的领域相关问题，并采用额外过滤策略保留高质量合成样本。借助这些自生成样本，模型在领域特定RAG任务上的表现得以提升。在覆盖两个模型规模和三个领域的11个数据集上的实验表明，SimRAG相比基线模型性能提升1.2%–8.6%。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) enhances the question-answering (QA) abilities of large language models (LLMs) by integrating external knowledge. However, adapting general-purpose RAG systems to specialized fields such as science and medicine poses unique challenges due to distribution shifts and limited access to domain-specific data. To tackle this, we propose SimRAG, a self-training approach that equips the LLM with joint capabilities of question answering and question generation for domain adaptation. Our method first fine-tunes the LLM on instruction-following, question-answering, and search-related data. Then, it prompts the same LLM to generate diverse domain-relevant questions from unlabeled corpora, with an additional filtering strategy to retain high-quality synthetic examples. By leveraging these self-generated synthetic examples, the LLM can improve their performance on domain-specific RAG tasks. Experiments on 11 datasets, spanning two backbone sizes and three domains, demonstrate that SimRAG outperforms baselines by 1.2\%--8.6\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。