针对高重复文档的检索评估难题,提出新型评测框架RARE。
RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
- 将文档拆解为原子事实,精准追踪信息冗余
- 在金融、法律、专利领域测试中,基线模型性能下降至5.0-27.9%
- 适合构建真实场景下的RAG系统评估,提升评测可靠性
现有问答评测基准通常假设文档之间差异显著、重叠极少,但现实中的检索增强生成(RAG)系统处理的是金融报告、法律条文和专利等高度冗余、文档间相似性极强的语料。这种差异导致评测失效:即使检索器提供了充分证据,也会因未考虑文档冗余而被低估。相反,标准基准表现优异的检索器在真实高相似性语料上泛化能力差。本文提出RARE(冗余感知检索评测框架),通过(i)将文档分解为原子事实以实现冗余精准追踪,(ii)结合CRRF改进大模型数据生成质量。传统生成常产生平庸结果,而CRRF分别评分各项质量指标并按排名融合,显著提升生成数据可靠性。将RARE应用于金融、法律与专利语料,构建出RedQA评测集。强基线模型在4跳任务下,PerfRecall@10从4-维通用维基的66.4%骤降至5.0%-27.9%,揭示了当前基准无法捕捉的鲁棒性缺陷。RARE使从业者能构建反映真实部署环境的领域专属RAG评测体系。
原文摘要 · Abstract (English)
Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity. This mismatch undermines evaluation validity: retrievers can be unfairly undervalued even when they retrieve documents that provide sufficient evidence, because redundancy across documents is not accounted for in evaluation. On the other hand, retrievers that perform well on standard benchmarks often generalize poorly to real-world corpora with highly similar and redundant documents. We present RARE (Redundancy-Aware Retrieval Evaluation), a framework for constructing realistic benchmarks by (i) decomposing documents into atomic facts to enable precise redundancy tracking and (ii) enhancing LLM-based data generation with CRRF. RAG benchmark data usually requires multiple quality criteria, but LLMs often yield trivial outputs. CRRF scores criteria separately and fuses decisions by rank, improving the reliability of generated data. Applying RARE to Finance, Legal, and Patent corpora, we introduce RedQA, where a strong retriever baseline drops from 66.4% PerfRecall@10 on 4-hop General-Wiki to 5.0-27.9% PerfRecall@10 at 4-hop depth, revealing robustness gaps that current benchmarks fail to capture. RARE enables practitioners to build domain-specific RAG evaluations that faithfully reflect real-world deployment conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。