用错数据集会误导RAG系统设计，本文提出更科学的评估与生成方法。
Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
- 通过标签化数据分类，精准识别RAG数据集类型
- 发现常用生成工具导致数据分布失衡，影响评估可信度
- 小模型微调后可高效生成高质量问答数据，适合本地化测试
检索增强生成（RAG）系统是大语言模型在工业界广泛应用的典型场景。尽管已有多种工具支持开发者构建系统，但在本地评估时使用反映实际应用场景的数据集仍面临技术挑战。现有解决方案从通用廉价（多数公开数据集）到专用昂贵（基于本地文档生成）不等。本文指出，使用通用问答数据集评估检索性能可能导致系统设计不优，且常见RAG数据生成工具易产生数据分布失衡。为此，我们提出基于标签化数据分类与目标标签驱动的数据生成策略。实验表明，微调后的小型语言模型可高效生成高质量问答数据集。这些发现对RAG开发中的‘知数据’阶段具有重要价值。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) systems are a widespread application of Large Language Models (LLMs) in the industry. While many tools exist empowering developers to build their own systems, measuring their performance locally, with datasets reflective of the system's use cases, is a technological challenge. Solutions to this problem range from non-specific and cheap (most public datasets) to specific and costly (generating data from local documents). In this paper, we show that using public question and answer (Q&A) datasets to assess retrieval performance can lead to non-optimal systems design, and that common tools for RAG dataset generation can lead to unbalanced data. We propose solutions to these issues based on the characterization of RAG datasets through labels and through label-targeted data generation. Finally, we show that fine-tuned small LLMs can efficiently generate Q&A datasets. We believe that these observations are invaluable to the know-your-data step of RAG systems development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。