提出细粒度评估框架，专测大模型在RAG中的知识利用能力
PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation
- 用占位符分离模型内知识与外部知识，精准评估模型表现
- 发现主流大模型在容错和忠实性上存在明显短板
- 适合研究RAG系统可靠性、追求模型可解释性的开发者
检索增强生成（RAG）通过融合外部知识提升大语言模型（LLM）性能，其核心在于模型能否基于查询与检索文档的组合生成回答。然而，现有基准多关注整体系统表现，很少评估模型本身的生成能力。当前评测侧重噪声鲁棒性，但缺乏对文档利用率的系统性细粒度分析。为此，我们提出 extit{Placeholder-RAG-Benchmark}，一个多层次细粒度评估框架，涵盖三个递进维度：（1）多层级过滤能力，（2）组合生成能力，（3）引用推理能力。通过创新的占位符方法，解耦模型参数化知识与外部知识的贡献，更清晰揭示大模型在RAG中的作用。实验表明，代表性大模型在生成能力上存在局限，尤其在错误容忍与上下文忠实性方面。该基准提供可复现的评估体系，助力构建更可靠高效的RAG系统。代码已开源：https://github.com/Alipay-Med/PRGB。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, where the LLM's ability to generate responses based on the combination of a given query and retrieved documents is crucial. However, most benchmarks focus on overall RAG system performance, rarely assessing LLM-specific capabilities. Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization. To this end, we introduce \textit{Placeholder-RAG-Benchmark}, a multi-level fine-grained benchmark, emphasizing the following progressive dimensions: (1) multi-level filtering abilities, (2) combination abilities, and (3) reference reasoning. To provide a more nuanced understanding of LLMs' roles in RAG systems, we formulate an innovative placeholder-based approach to decouple the contributions of the LLM's parametric knowledge and the external knowledge. Experiments demonstrate the limitations of representative LLMs in the RAG system's generation capabilities, particularly in error resilience and context faithfulness. Our benchmark provides a reproducible framework for developing more reliable and efficient RAG systems. Our code is available in https://github.com/Alipay-Med/PRGB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。