arXiv:2510.07098cs.CL2025-10被引 4

轻量级表格问答框架,用自然语言叙述提升小模型表现

TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription

  • 小视觉模型输出文字+自然语言描述,由大语言模型统一推理
  • 在公开数据集和新数据集上性能超越单一大模型,计算成本更低
  • 适合移动端部署,尤其擅长需要多步推理的复杂表格问题

表格视觉问答(Table VQA)通常依赖大型视觉-语言模型(VLM),但这类模型若不规模庞大则难以捕捉细节,且大规模模型在移动设备上部署成本过高。另一种轻量方案是让小型VLM执行OCR,再交由大语言模型(LLM)处理结构化输出(如Markdown表格),但此类表示并非为LLM优化,仍存在显著误差。本文提出TALENT(基于增强语言增强自然文本转录的表格问答),采用双重表征机制:通过小型VLM生成原始文字与自然语言叙述,并将二者与问题一同输入LLM进行推理。该方法将表格问答重构为以LLM为核心的多模态推理任务,使VLM仅承担感知与叙述功能。同时,我们构建了更具挑战性的ReTabVQA数据集,要求对表格图像进行多步定量推理。实验表明,TALENT使小型VLM-LLM组合在多个公开数据集及ReTabVQA上的表现达到甚至超越单一大型VLM,且计算开销显著降低。

原文摘要 · Abstract (English)

Table Visual Question Answering (Table VQA) is typically addressed by large vision-language models (VLMs). While such models can answer directly from images, they often miss fine-grained details unless scaled to very large sizes, which are computationally prohibitive, especially for mobile deployment. A lighter alternative is to have a small VLM perform OCR and then use a large language model (LLM) to reason over structured outputs such as Markdown tables. However, these representations are not naturally optimized for LLMs and still introduce substantial errors. We propose TALENT (Table VQA via Augmented Language-Enhanced Natural-text Transcription), a lightweight framework that leverages dual representations of tables. TALENT prompts a small VLM to produce both OCR text and natural language narration, then combines them with the question for reasoning by an LLM. This reframes Table VQA as an LLM-centric multimodal reasoning task, where the VLM serves as a perception-narration module rather than a monolithic solver. Additionally, we construct ReTabVQA, a more challenging Table VQA dataset requiring multi-step quantitative reasoning over table images. Experiments show that TALENT enables a small VLM-LLM combination to match or surpass a single large VLM at significantly lower computational cost on both public datasets and ReTabVQA.

表格问答轻量模型多模态推理自然语言叙述

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。