首次系统分析RAG部署的三大关键决策，揭示其效果受任务类型和模型特性影响。
Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
- 通过三类模型、六数据集实验，研究是否用RAG、查多少信息、如何融合知识
- 问答任务最优检索5-10篇文档，代码生成需按场景定制，错误率最高达12.6%
- 代码生成靠提示词提升明显，问答任务改进有限，需因任务设计策略
检索增强生成（RAG）已成为提升大语言模型能力的关键技术。然而，实践者在部署RAG时面临重大挑战。现有研究多聚焦算法创新，但对决定RAG成败的根本工程权衡缺乏系统理解。本文首次全面研究三个通用的RAG部署决策：是否部署RAG、检索信息量大小、如何有效整合检索知识。通过在三种LLM和六个涵盖问答与代码生成任务的数据集上进行系统实验，揭示关键发现：(1) RAG部署必须高度选择性，即使文档完美，可变召回阈值和失败模式仍会影响高达12.6%的样本；(2) 最优检索量具任务依赖性，问答任务呈现普遍规律（5-10篇最优），而代码生成需场景特异性优化；(3) 知识整合效果取决于任务与模型特征，代码生成显著受益于提示方法，问答任务改善甚微。结果表明，通用RAG策略不足为恃。高效RAG系统需基于任务特征与模型能力进行上下文感知的设计。本研究为从业者提供实证指导，并建立原则性RAG部署的基础洞见。代码、数据与成果已公开于https://github.com/ShengmingZ/RAG_Benchmark_Code_QA。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has emerged as a critical technique for enhancing large language model (LLM) capabilities. However, practitioners face significant challenges when making RAG deployment decisions. While existing research prioritizes algorithmic innovations, a systematic gap persists in understanding fundamental engineering trade-offs that determine RAG success. We present the first comprehensive study of three universal RAG deployment decisions: whether to deploy RAG, how much information to retrieve, and how to integrate retrieved knowledge effectively. Through systematic experiments across three LLMs and six datasets spanning question answering and code generation tasks, we reveal critical insights: (1) RAG deployment must be highly selective, with variable recall thresholds and failure modes affecting up to 12.6\% of samples even with perfect documents. (2) Optimal retrieval volume exhibits task-dependent behavior QA tasks show universal patterns (5-10 documents optimal) while code generation requires scenario-specific optimization. (3) Knowledge integration effectiveness depends on task and model characteristics, with code generation benefiting significantly from prompting methods while question answering shows minimal improvement. These findings demonstrate that universal RAG strategies prove inadequate. Effective RAG systems require context-aware design decisions based on task characteristics and model capabilities. Our analysis provides evidence-based guidance for practitioners and establishes foundational insights for principled RAG deployment. Our code, data and artifacts are publicly available at https://github.com/ShengmingZ/RAG_Benchmark_Code_QA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。