arXiv:2501.13042cs.CL2025-01被引 17

针对科学表格理解难题,提出动态分辨率多模态模型与专用数据集。

Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning

  • 构建动态输入分辨率的多模态模型,直接处理科学表格图像。
  • 52K专业表格数据训练优于150K通用表格数据,证明质量重于数量。
  • 专为数值推理设计的基准测试,适合科研与医学领域应用。

近期大语言模型虽提升了表格理解能力,但依赖将表格转为文本序列。多模态大模型虽可直接处理图像,却受限于固定图像分辨率和不足的数值推理能力。本文提出一个完整的多模态科学表格理解与推理框架,包含三个核心组件:(1) MMSci-Pre,包含52,000个科学表格结构识别样本的领域专用预训练数据集;(2) MMSci-Ins,涵盖3种表格任务、共12,000个样本的指令微调数据集;(3) MMSci-Eval,专为评估数值推理能力设计的基准,含3,114个测试样本。大量实验表明,使用52,000个科学表格图像的领域专用方法,在性能上显著优于150,000个通用领域表格,凸显数据质量的重要性。所提基于表格的多模态模型在动态输入分辨率下,大幅提升了通用表格理解与数值推理能力,并在未见数据集上表现良好。代码与数据已公开于https://github.com/Bernard-Yang/MMSci_Table。

原文摘要 · Abstract (English)

Recent large language models (LLMs) have advanced table understanding capabilities but rely on converting tables into text sequences. While multimodal large language models (MLLMs) enable direct visual processing, they face limitations in handling scientific tables due to fixed input image resolutions and insufficient numerical reasoning capabilities. We present a comprehensive framework for multimodal scientific table understanding and reasoning with dynamic input image resolutions. Our framework consists of three key components: (1) MMSci-Pre, a domain-specific table structure learning dataset of 52K scientific table structure recognition samples, (2) MMSci-Ins, an instruction tuning dataset with 12K samples across three table-based tasks, and (3) MMSci-Eval, a benchmark with 3,114 testing samples specifically designed to evaluate numerical reasoning capabilities. Extensive experiments demonstrate that our domain-specific approach with 52K scientific table images achieves superior performance compared to 150K general-domain tables, highlighting the importance of data quality over quantity. Our proposed table-based MLLMs with dynamic input resolutions show significant improvements in both general table understanding and numerical reasoning capabilities, with strong generalisation to held-out datasets. Our code and data are publicly available at https://github.com/Bernard-Yang/MMSci_Table.

科学表格多模态数值推理数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。