构建可扩展的跨文本与表格多跳问答基准,支持复杂推理和聚合操作。
SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables
- 自动构建大规模问答数据集,仅需四分之一人工标注时间
- 生成包含深度多跳、聚合与分组的高保真问题,覆盖真实分析场景
- 揭示当前模型在跨模态推理中的严重缺陷,适合评估高级推理能力
真实世界的文本-表格问答任务需要模型能够跨长文本和源表进行多跳推理,并执行聚合等复杂操作。然而现有基准规模小、人工标注易出错,且问题浅显,极少要求超过两跳或涉及聚合、分组等高级分析操作。本文提出 SPARTA,一个端到端的自动化构建框架,可生成大规模高质量表格-文本问答基准,人工验证仅需 HybridQA 的四分之一时间。该框架首先通过从非结构化段落中提取原子事实,丰富原始表格以构建参考事实库;随后合成嵌套查询,其嵌套谓词数量匹配目标跳数。为确保每个 SQL 可执行且自然语言表述流畅,提出两种新方法:基于溯源的精炼(重写返回非空结果的语法正确查询),以及真实结构约束(限定生成为查询图的后序遍历)。最终生成数千对高保真问答对,涵盖聚合、分组与深层多跳推理。在 SPARTA 基准上,现有最优模型在 HybridQA 上超过 70 F1 或在 OTT-QA 上超过 50 F1 的表现,下降超过 30 F1 点,暴露出当前跨模态推理模型的根本性不足。相关基准、代码与基线模型已开源。
原文摘要 · Abstract (English)
Real-world Table-Text question answering (QA) tasks require models that can reason across long text and source tables, traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated - and therefore error-prone - and contain shallow questions that seldom demand more than two hops or invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, an end-to-end construction framework that automatically generates large-scale Table-Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQL statement is executable and that its verbalization yields a fluent, human-sounding question, we propose two novel techniques: provenance-based refinement, which rewrites any syntactically valid query that returns a non-empty result, and realistic-structure enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of high-fidelity question-answer pairs covering aggregations, grouping, and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 F1 on HybridQA or over 50 F1 on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning. Our benchmark, construction code, and baseline models are available at https://github.com/pshlego/SPARTA/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。