构建多文档多模态科学问答数据集，推动复杂信息推理研究
DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections
- 基于11种科学推理概念，用LLM生成跨文档多跳问答对
- 包含11,379个实例，融合文本、表格与版式线索支持跨文档推理
- 适合研究长上下文理解与多源证据整合的模型开发者
尽管大语言模型进展迅速，现有问答评测仍忽视真实科研信息检索的核心挑战：在多文档与异构格式中整合分散的多模态证据。现有评测范围狭窄，依赖单一文本和短跨度推理，难以反映真实信息搜索的复杂性。我们提出DocHop-QA，一个包含11,379个实例的基准数据集，用于评估多模态、多文档、多跳科学问答。数据源自公开的PubMed文章，涵盖文本段落、表格和版式线索，支持无显式超链接的跨文档推理。为实现真实问答的规模化构建，我们开发了基于11种科学推理概念的LLM驱动生成流程，生成多样且连贯的问答对。为凸显数据集的实用性与通用性，我们提出任务驱动的评估框架，覆盖四种场景：生成式回答、多模态证据融合与结构化索引预测。实验表明，当前模型难以应对DocHop-QA的长上下文与多证据需求，确立其作为下一代科学问答系统推进的严格测试平台。
原文摘要 · Abstract (English)
Despite rapid progress in large language models (LLMs), current QA benchmarks still overlook the core challenge of real-world scientific information seeking: synthesizing multimodal evidence scattered across multiple documents and structural formats. Existing QA benchmarks remain narrow in scope, relying on unimodal text and short-span reasoning that fail to capture the complexity of real information seeking. We introduce DocHop-QA, a benchmark of 11,379 instances for evaluating multimodal, multi-document, multi-hop scientific QA. Built from publicly available PubMed articles, DocHop-QA incorporates textual passages, tables, and layout cues, enabling cross-document inference without explicit hyperlinks. To scale realistic QA construction, we develop an LLM-driven generation pipeline grounded in 11 scientific reasoning concepts, producing diverse and coherent question-answer pairs. To highlight the utility and versatility of the dataset, we propose a task-driven evaluation framework spanning four settings, including generative answering, multimodal evidence integration, and structured index prediction. Experiments show that current models struggle with the long-context and multi-evidence demands of DocHop-QA, establishing it as a rigorous testbed for advancing next-generation scientific QA systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。