构建科学表格问答新基准，测试模型多步推理与执行能力
SciTaRC: A Plan-Annotated Scientific Tabular QA Benchmark for Language Reasoning and Complex Computation
- 为科学表格问答设计带推理路径的标注数据集
- 主流模型在65.5%的问题上失效，暴露执行瓶颈
- 即使给出正确步骤，模型仍难完成复杂计算链
我们提出SciTaRC，一个由专家构建的科学表格问答基准，旨在评估复合型、多步推理能力。为实现细粒度诊断分析，SciTaRC为每道题配以人工构造的推理计划和明确的复杂度指标。最先进模型在至少23%的问题上失败，而像Llama-3.3-70B这类开源大模型在65.5%的问题上崩溃。错误分析显示，在零样本设置下，失败主要源于问题理解不足，模型误读科学查询并得出错误推理目标。为检验克服此差距是否足够，我们利用结构化计划将策略制定与执行解耦。令人意外的是，提供理想化的逐步计划仅带来有限提升，无法消除性能差距。这揭示出显著的执行瓶颈：无论是自然语言还是代码方法，都难以可靠完成对结构化数据的长时程计算链条。最终，SciTaRC成为研究科学表格推理中规划与执行能力的严格诊断平台。
原文摘要 · Abstract (English)
We introduce SciTaRC, an expert-authored benchmark for question answering over scientific tables that targets composite, multi-step reasoning. To enable fine-grained diagnostic analysis beyond end-task accuracy, SciTaRC pairs each question with a manually constructed reasoning plan and explicit complexity metrics. State-of-the-art models fail on at least 23% of these questions, while highly capable open-weight models like Llama-3.3-70B collapse on 65.5% of the benchmark. Error analysis shows that, in zero-shot settings, failures are driven primarily by question comprehension, where models misinterpret the scientific query and derive the wrong reasoning objective. To determine whether overcoming this gap is sufficient, we use the structured plans to decouple strategy formulation from execution. Surprisingly, providing oracle step-by-step plans yields only limited gains and fails to eliminate the performance gap. This reveals a substantial execution bottleneck: both natural language and code-based methods struggle to reliably carry out long-horizon computational chains over structured data. Ultimately, SciTaRC serves as a rigorous diagnostic testbed for studying both planning and execution in scientific table reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。