arXiv:2609.05141cs.AI2026-09

构建首个面向科研阅读流程的多模态文档理解基准,评估模型在真实场景下的综合能力。

SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding

论文配图:SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding
图 1 · 摘自论文原文
  • 以科研助手工作流为核心设计评测任务,覆盖文本、公式、图表等多模态信息
  • 最强模型仅达62.6分,暴露文档感知与证据溯源等关键短板
  • 提出可训练的证据图结构,支持高效标注与强化学习微调

科研论文要求模型协同理解文本、公式、图表、表格、代码和数据集,并保留证据来源。现有基准多孤立评估各项能力,难以判断多模态模型是否支持真实的科研阅读流程。我们提出SciDocBench,一个以工作流为中心的科学文档理解基准,包含124个专家撰写且难度筛选的问题,分为7类研究助理能力与19个子任务,覆盖5个科学领域。每个问题在英语或中文提问,搭配全图先行或交错排布的文档表示下生成4种条件,共496个评估实例,实现可控分析。最强系统得分仅62.6/100,显著暴露文档感知、证据定位、验证及跨文档推理等方面的不足。为将诊断转化为可扩展的训练信号,我们引入SciDocIR——一种保留科学文档对象、布局与交叉引用关系及出处的类型化证据图结构。基于SciDocIR,构建了涵盖约1.5万条监督微调样本和8千条强化学习样本的SciDocDataset,覆盖14个可验证子任务。三者共同构成从评估到训练的完整框架。项目主页见:https://github.com/InternLM/SciDocBench。

原文摘要 · Abstract (English)

Scientific papers require models to reason jointly over text, equations, figures, tables, code, and datasets while preserving the provenance of supporting evidence. Existing benchmarks typically evaluate these capabilities in isolation, leaving unclear whether multimodal models can support realistic scientific-reading workflows. We introduce SciDocBench, a workflow-centered benchmark for scientific document understanding. It contains 124 expert-authored and difficulty-screened questions organized into seven research-assistant capability groups and 19 subtasks across five scientific domains. Each question is instantiated under four matched conditions combining English or Chinese questions with all-images-first or interleaved document representations, yielding 496 evaluation instances for controlled analysis. The strongest evaluated system achieves only 62.6/100, with pronounced weaknesses in document perception, evidence grounding, verification, and cross-document reasoning. To translate these diagnostics into scalable training signals, we introduce SciDocIR, a typed evidence-graph representation that preserves scientific document objects, layout and cross-reference relations, and provenance. Building on SciDocIR, we construct SciDocDataset, comprising approximately 15K supervised fine-tuning samples and 8K reinforcement-learning samples across 14 verifiable subtasks. Together, SciDocBench, SciDocIR, and SciDocDataset form an evaluation-to-training framework for diagnosing and improving scientific-document assistants. The project page is available at https://github.com/InternLM/SciDocBench.

文档理解多模态科研助手评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。