用多模态大模型从海量表格图像中高效找图并理解答案
Efficient Table Retrieval and Understanding with Multimodal Large Language Models
- 先用视觉文本模型粗筛表格，再用大模型精排候选
- 在8个基准上检索召回率提升7.0%，答案准确率提高6.1%
- 适合需要从扫描件/图片中提取信息的金融、文档场景
表格数据常以图像形式存在于财务报告、手写记录和文档扫描等真实场景中，其结构与视觉复杂性给机器理解带来挑战。尽管多模态大语言模型（MLLMs）在表格理解方面表现良好，但通常假设目标表格已知。更实际的场景是从大规模表格图像集合中定位并推理出相关表格以回答用户问题。为此，我们提出TabRAG框架，使MLLMs能对大规模表格图像集进行问答。该方法首先通过联合训练的视觉-文本基础模型检索候选表格，再利用MLLM进行细粒度重排序，最后由MLLM对选定表格进行推理生成答案。在包含88,161个训练样本和9,819个测试样本的新型数据集上，涵盖8个基准与48,504张唯一表格的实验表明，本框架在检索召回率上比现有方法提升7.0%，答案准确率提升6.1%，为现实世界表格理解任务提供实用解决方案。
原文摘要 · Abstract (English)
Tabular data is frequently captured in image form across a wide range of real-world scenarios such as financial reports, handwritten records, and document scans. These visual representations pose unique challenges for machine understanding, as they combine both structural and visual complexities. While recent advances in Multimodal Large Language Models (MLLMs) show promising results in table understanding, they typically assume the relevant table is readily available. However, a more practical scenario involves identifying and reasoning over relevant tables from large-scale collections to answer user queries. To address this gap, we propose TabRAG, a framework that enables MLLMs to answer queries over large collections of table images. Our approach first retrieves candidate tables using jointly trained visual-text foundation models, then leverages MLLMs to perform fine-grained reranking of these candidates, and finally employs MLLMs to reason over the selected tables for answer generation. Through extensive experiments on a newly constructed dataset comprising 88,161 training and 9,819 testing samples across 8 benchmarks with 48,504 unique tables, we demonstrate that our framework significantly outperforms existing methods by 7.0% in retrieval recall and 6.1% in answer accuracy, offering a practical solution for real-world table understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。