arXiv:2603.20034cs.IRcs.AI2026-03被引 3

构建跨任务与领域的信息覆盖评估基准,解决传统指标忽略重复信息的问题。

CoverageBench: Evaluating Information Coverage across Tasks and Domains

  • 基于现有数据集构建覆盖多领域、多任务的评估套件
  • 提出衡量检索结果是否涵盖全部相关知识的新评估标准
  • 适合研究RAG系统或信息检索覆盖能力的学者使用

我们希望衡量即兴检索算法的信息覆盖度,即检索结果涵盖可用相关资讯的比例。在检索增强生成(RAG)系统中,信息覆盖至关重要。传统指标如精确率和召回率随着更多相关文档被检索而得分提高,但这些指标不考虑文档间信息重复性——即使已包含某信息,额外检索相同内容仍被奖励。排名偏置精度(RBP)、归一化折损累积收益(nDCG)和平均精度均值(MAP)同样存在此问题。虽然部分网页搜索中的多样性排序测试集已引入覆盖概念,但缺乏统一框架。本文构建了一套来自现有集合的信息覆盖评估套件,提供跨多个类型和任务的统一测试平台。所有主题、知识点、相关性标注及基线排序均发布于Hugging Face Datasets,并附有访问公开文档集合的说明。

原文摘要 · Abstract (English)

We wish to measure the information coverage of an ad hoc retrieval algorithm, that is, how much of the range of available relevant information is covered by the search results. Information coverage is a central aspect for retrieval, especially when the retrieval system is integrated with generative models in a retrieval-augmented generation (RAG) system. The classic metrics for ad hoc retrieval, precision and recall, reward a system as more and more relevant documents are retrieved. However, since relevance in ad hoc test collections is defined for a document without any relation to other documents that might contain the same information, high recall is sufficient but not necessary to ensure coverage. The same is true for other metrics such as rank-biased precision (RBP), normalized discounted cumulative gain (nDCG), and mean average precision (MAP). Test collections developed around the notion of diversity ranking in web search incorporate multiple aspects that support a concept of coverage in the web domain. In this work, we construct a suite of collections for evaluating information coverage from existing collections. This suite offers researchers a unified testbed spanning multiple genres and tasks. All topics, nuggets, relevance labels, and baseline rankings are released on Hugging Face Datasets, along with instructions for accessing the publicly available document collections.

信息覆盖检索评估RAG

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。