自动测试RAG系统，发现生成与检索的协同故障。
RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models
- 按覆盖度生成测试用例，聚焦复杂文本和无效查询。
- 24种配置下检测出2.16万次失败，比基线多6.6%。
- 适合评估RAG部署前的可靠性，尤其关注检索-生成交互。
检索增强生成（RAG）使大语言模型能利用外部领域知识，但其可靠性取决于生成模型、嵌入模型、检索机制和提示构造策略之间的交互。本文提出RagTester，一种自动化端到端测试RAG系统的方案。RagTester自动生成检索文档、测试输入和期望输出；执行测试；并使用大语言模型作为评判者评估结果。其测试生成策略针对复杂段落、无支持查询和文档覆盖率。我们在8个LLM和6个嵌入模型组合的24种配置上评估RagTester，与基线测试生成器对比，共执行72,000次测试，检测到21,633次失败，比基线多6.6%，在20/24种配置中表现更优。失败类型包括检索不准、答案无依据、未充分使用检索上下文以及理解复杂段落困难。结果表明，以覆盖率为导向的测试生成能有效暴露检索与生成组件间交互导致的缺陷，支持部署前的RAG配置评估。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。