用合成数据+模型融合，让小数据下检索更准
REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Models
- 用已有文档生成合成数据，再融合多个嵌入模型微调
- 在旅游数据集上召回率提升5.76%，问答任务提升超6%
- 适合缺乏标注数据但需精准检索的工业场景
检索增强生成（RAG）常用于问答任务，依赖预训练嵌入模型从向量库中检索相关文档。若检索结果不准确，大语言模型生成的答案可能出错或产生幻觉。尽管预训练嵌入模型已取得进展，但在新领域适配仍具挑战。微调是潜在解决方案，但工业场景常缺乏足够的微调数据。为此，我们提出REFINE：通过现有文档生成合成数据，并采用模型融合方式微调嵌入模型，以提升新领域的检索性能，同时保持跨域能力。我们在SQUAD、RAG-12000及自有TOURISM数据集上进行了实验。结果表明，仅使用所提数据增强技术的标准微调即优于原始预训练模型；结合模型融合后，该方法在TOURISM数据集上召回率提升5.76%，在SQUAD和RAG-12000上分别提升6.58%和0.32%。
原文摘要 · Abstract (English)
Retrieval augmented generation (RAG) pipelines are commonly used in tasks such as question-answering (QA), relying on retrieving relevant documents from a vector store computed using a pretrained embedding model. However, if the retrieved context is inaccurate, the answers generated using the large language model (LLM) may contain errors or hallucinations. Although pretrained embedding models have advanced, adapting them to new domains remains challenging. Fine-tuning is a potential solution, but industry settings often lack the necessary fine-tuning data. To address these challenges, we propose REFINE, a novel technique that generates synthetic data from available documents and then uses a model fusion approach to fine-tune embeddings for improved retrieval performance in new domains, while preserving out-of-domain capability. We conducted experiments on the two public datasets: SQUAD and RAG-12000 and a proprietary TOURISM dataset. Results demonstrate that even the standard fine-tuning with the proposed data augmentation technique outperforms the vanilla pretrained model. Furthermore, when combined with model fusion, the proposed approach achieves superior performance, with a 5.76% improvement in recall on the TOURISM dataset, and 6.58 % and 0.32% enhancement on SQUAD and RAG-12000 respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。