arXiv:2506.04088cs.LGcs.AI2025-06NeurIPS被引 14

用结构化信息辅助图像表格推理,提升模型表现。

Multimodal Tabular Reasoning with Privileged Structured Information

  • 利用训练时的结构信息桥接视觉与文本表示
  • 仅用9000条数据即达当前最优,性能提升7.2%
  • 适合需要图像表格推理的现实场景应用

表格推理涉及对结构化表格数据进行多步信息提取与逻辑推断。尽管近期研究利用大语言模型(LLMs)处理表格推理,但真实场景中表格常以图像形式存在,缺乏高质量文本表示。本文针对从表格图像进行推理的任务,利用训练阶段可获得的结构化信息来增强多模态大语言模型(MLLMs)。核心挑战在于如何准确对齐结构化信息与视觉表征,并克服输入模态差异有效迁移结构化推理能力。为此,我们提出{ extsc{Turbo}}框架——一种基于结构感知推理路径生成器(基于DeepSeek-R1)的多模态表格推理方法。该框架通过生成并筛选优势推理路径,持续提升模型的表格推理能力。实验表明,在仅使用9000条数据的情况下,{ extsc{Turbo}}在多个数据集上均达到领先性能,相比此前最优方法提升7.2%。

原文摘要 · Abstract (English)

Tabular reasoning involves multi-step information extraction and logical inference over tabular data. While recent advances have leveraged large language models (LLMs) for reasoning over structured tables, such high-quality textual representations are often unavailable in real-world settings, where tables typically appear as images. In this paper, we tackle the task of tabular reasoning from table images, leveraging privileged structured information available during training to enhance multimodal large language models (MLLMs). The key challenges lie in the complexity of accurately aligning structured information with visual representations, and in effectively transferring structured reasoning skills to MLLMs despite the input modality gap. To address these, we introduce TabUlar Reasoning with Bridged infOrmation ({\sc Turbo}), a new framework for multimodal tabular reasoning with privileged structured tables. {\sc Turbo} benefits from a structure-aware reasoning trace generator based on DeepSeek-R1, contributing to high-quality modality-bridged data. On this basis, {\sc Turbo} repeatedly generates and selects the advantageous reasoning paths, further enhancing the model's tabular reasoning ability. Experimental results demonstrate that, with limited ($9$k) data, {\sc Turbo} achieves state-of-the-art performance ($+7.2\%$ vs. previous SOTA) across multiple datasets.

表格推理多模态结构信息图像表格

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。