梳理RAG系统常见错误类型并提供解决方法
Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems
- 提出RAG错误类型新分类体系
- 构建带错误标注的实证数据集
- 开发可落地的自动化评估工具
检索增强生成（RAG）是利用外部知识库构建大模型问答系统的一种主流方法。由于真实RAG系统的复杂性，错误可能由多种因素引起。理解实际中可能出现的错误范围对系统稳健部署至关重要。本文提出一种适用于现实RAG系统的错误类型新分类体系，包含各类错误的实例及应对建议，并构建了一个由人工标注错误类型的错误RAG响应数据集。此外，我们设计了一种与该分类体系对齐的自动化评估方法，可在开发过程中用于追踪和解决错误。相关代码与数据已开源。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) is a prevalent approach for building LLM-based question-answering systems that can take advantage of external knowledge databases. Due to the complexity of real-world RAG systems, there are many potential causes for erroneous outputs. Understanding the range of errors that can occur in practice is crucial for robust deployment. We present a new taxonomy of the error types that can occur in realistic RAG systems, examples of each, and practical advice for addressing them. Additionally, we curate a dataset of erroneous RAG responses annotated by error types. We then propose an auto-evaluation method aligned with our taxonomy that can be used in practice to track and address errors during development. Code and data are available at https://github.com/layer6ai-labs/rag-error-classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。