用符号推理增强表格与视觉的多模态理解能力
Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning
- 引入代码驱动的神经符号机制，动态处理表格结构和信息
- 在8个数据集上平均提升10%准确率，超越主流商业模型
- 适合需要精确表格推理的科研与工业场景
多模态大语言模型在图像与文本等模态中展现出强大推理能力，但表格数据这一关键现实模态仍缺乏充分探索。本文聚焦表格-视觉多模态理解（TVMU）任务，识别出三大挑战：（1）表格结构高度可变且数据不完整；（2）特征间存在隐含复杂依赖；（3）下游任务求解流程差异显著。为此，提出“思考表格”（TWT）框架，采用程序辅助的代码化神经符号推理机制，通过与外部环境交互实现信息抽取与元素建模等关键操作。在八个代表性数据集上的实验表明，TWT平均准确率较现有基线提升10%，性能达到甚至超过部分专有商用先进大模型水平。代码与模型已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10\% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。