提出RARE框架，评估RAG系统在真实噪声下的稳定性。
RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems
- 构建知识图谱驱动的自动问答生成流水线，实现无人工干预的多跳问题合成。
- 基于527篇时效性文献构建48295个动态演化的问题集，覆盖金融、经济与政策领域。
- 发现RAG系统对扰动敏感，多跳查询鲁棒性显著低于单跳查询，适用于可靠性研究者。
检索增强生成（RAG）提升了回答的时效性和事实准确性，但现有评估很少测试系统在现实噪声下的表现，如内部与外部检索内容冲突或事实快速变化的情况。本文提出检索感知鲁棒性评估框架RARE，包含大规模基准数据集和联合压力测试机制，针对动态、时敏语料库中的查询与文档扰动进行综合评估。RARE的核心是知识图谱驱动的合成流水线（RARE-Get），可从定制语料中自动提取单跳与多跳关系，并生成多层级问题集而无需人工干预。基于此，我们构建了涵盖527篇专家级时敏文献及48295个随源变化而演化的问答数据集（RARE-Set）。为量化系统韧性，我们定义了检索条件下的鲁棒性度量（RARE-Met），衡量模型在查询、文档或真实检索结果被系统性扰动时保持正确性或恢复能力的表现。实验结果表明，RAG系统对扰动异常敏感，且在所有领域中，多跳查询的鲁棒性均显著低于单跳查询。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enhances recency and factuality in answers. However, existing evaluations rarely test how well these systems cope with real-world noise, conflicting between internal and external retrieved contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness Evaluation (RARE), a unified framework and large-scale benchmark that jointly stress-tests query and document perturbations over dynamic, time-sensitive corpora. One of the central features of RARE is a knowledge-graph-driven synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop relations from the customized corpus and generates multi-level question sets without manual intervention. Leveraging this pipeline, we construct a dataset (RARE-Set) spanning 527 expert-level time-sensitive finance, economics, and policy documents and 48295 questions whose distribution evolves as the underlying sources change. To quantify resilience, we formalize retrieval-conditioned robustness metrics (RARE-Met) that capture a model's ability to remain correct or recover when queries, documents, or real-world retrieval results are systematically altered. Our findings reveal that RAG systems are unexpectedly sensitive to perturbations. Moreover, they consistently demonstrate lower robustness on multi-hop queries compared to single-hop queries across all domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。