剖析RAG系统在不同设置下的敏感性与稳定性，揭示评估短板。
A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation
- 控制变量实验，测试56次运行中多个组件的影响
- 小块大小导致答案丢失，广度检索引发性能波动
- 建议综合多阶段失败分析，而非仅看最终准确率
检索增强生成（RAG）系统常以最终答案准确率评估，但其失效可能源于预处理、检索、上下文打包或生成阶段。本文通过56次受控实验，分析了块大小、检索深度（top k）、基于嵌入的重排序、概率性检索噪声及重复种子运行对检索、上下文打包和生成行为的影响。使用固定500个问题的QA子集映射至20,958个唯一语料上下文，既评估最终答案指标，也分析中间失败模式。结果显示，更广的检索设置提升了检索指标，但下游精确匹配和F1分数呈现非单调变化；小块大小引发预处理导致的答案丢失，检索污染造成渐进式退化，广度检索下方差显著升高。这些发现表明，RAG评估应包含敏感性、鲁棒性、稳定性及多阶段故障分析，而不能仅依赖最终答案准确率。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy, even though their failures can originate from preprocessing, retrieval, context packing, or generation. This paper presents a controlled empirical study of RAG sensitivity, robustness, and stability across 56 experimental runs. We evaluate how chunk size, retrieval depth (top k), embedding-based reranking, probabilistic retrieval noise, and repeated seeded runs affect retrieval, context packing, and generation behavior. Using a fixed 500-question QA subset mapped to 20,958 unique corpus contexts, we analyze both final answer metrics and intermediate failure modes. Across these experiments, retrieval-oriented metrics improved under broader retrieval settings, while downstream exact-match and F1 scores often behaved non-monotonically. We also observe preprocessing-induced answer loss under smaller chunk sizes, progressive degradation under retrieval corruption, and higher observed variance in broader retrieval regimes. These findings suggest that RAG evaluation should include sensitivity, robustness, stability, and multi-stage failure analysis rather than relying only on final answer accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。