arXiv:2607.03028cs.IR2026-07

构建首个覆盖五类异构数据源的问答检索基准,挑战多源信息融合能力。

HETERQA: Benchmarking Record Retrieval over Multiple Heterogeneous Sources

论文配图:HETERQA: Benchmarking Record Retrieval over Multiple Heterogeneous Sources
图 1 · 摘自论文原文
  • 基于真实业务数据,以答案驱动方式构建多源联合检索框架
  • 混合检索法在Recall@10上表现最优,但整体性能仍远未饱和
  • 适合研究多源信息融合、智能检索与自回归生成式检索的学者

在新兴系统(如社交媒体和电商平台)中,数据记录常来自异构来源,如关系表、文本文档、图像库、空间数据库和知识图谱。因此,问答任务中的目标记录检索需联合利用这些异构来源。然而,现有基准大多仅基于单一来源,仅有少数近期工作涉及两到三个来源。为缓解此问题,我们提出HETERQA,一个包含857个问答对的综合性基准,用于五类异构来源上的记录检索。HETERQA以Yelp商家记录为基础,每条记录由多个来源支撑。该基准采用答案驱动构建:先通过记录字段约束初始化候选记录,再经由异构来源扩充,并在必要来源间交叉验证后保留自然语言问题。我们通过矛盾检测与人工验证评估基准可靠性,并在相同指标下评估稀疏、稠密、混合、晚期交互及代理式检索器。结果表明,HETERQA极具挑战性:混合检索在Recall@10上表现最佳,Self-RAG在MRR@10上最优,所有方法均未达到基准上限。这说明HETERQA是异构来源记录检索的有效测试平台,未来检索方法仍有巨大提升空间。数据集与代码已公开于https://huggingface.co/datasets/hanchang02/HeterQA 和 https://github.com/hanchang02/HeterQA。

原文摘要 · Abstract (English)

In emerging systems (e.g., social media and e-commerce platforms), data records are often drawn from heterogeneous sources, such as relational tables, text documents, image repositories, spatial databases, and knowledge graphs. Accordingly, retrieving target records for question-answering (QA) tasks requires us to jointly exploit these heterogeneous sources. However, most existing benchmarks are constructed from individual sources, and only a very few recent benchmarks have considered two or three sources. To alleviate this issue, we introduce HETERQA, a comprehensive benchmark with 857 QA pairs for record retrieval over five heterogeneous sources. HETERQA instantiates this setting with Yelp business records, each of which is grounded by multiple sources. We build HETERQA in an answer-driven manner: candidate records are first initialized with record-field constraints, then enriched through heterogeneous sources, and finally cross-verified across required sources before the natural-language question is retained. We validate the benchmark through contradiction detection and human validation, and further evaluate sparse, dense, hybrid, late-interaction, and agentic retrievers under the same metrics. The results show that HETERQA is challenging: hybrid retrieval achieves the strongest Recall@10, Self-RAG achieves the best MRR@10, and all evaluated methods remain far from saturating the benchmark. These findings indicate that HETERQA provides an effective testbed for record retrieval over heterogeneous sources and leaves substantial room for future retrieval methods. The benchmark dataset and source code are publicly available at https://huggingface.co/datasets/hanchang02/HeterQA and https://github.com/hanchang02/HeterQA, respectively.

多源检索问答系统异构数据基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。