新基准揭示视觉语言模型在真实表格上的两大致命缺陷。
Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tables
- 构建多语言带噪声的表格问答数据集，模拟真实文档场景。
- 顶级模型在视觉噪声下性能下降超35%，跨语言推理能力严重不足。
- 适合关注模型鲁棒性与多语言泛化能力的研究者。
当前视觉语言模型（VLMs）的优异表现主要基于未能反映现实复杂性的基准测试。现有表格问答数据集如WikiTableQuestions和FinQA几乎全是英文且格式完美，导致研究与实际应用间存在巨大差距。为此，我们提出新基准MirageTVQA，包含近60,000个跨24种语言的问答对，挑战模型在多语言与视觉噪声下的表现，模拟扫描文档的真实情况。对主流VLMs的评估显示，面对视觉噪声时性能下降超过35%（最佳模型），且普遍存在英语优先偏差，跨语言推理能力严重受限。MirageTVQA为衡量和推动更鲁棒的表格推理模型发展提供了关键工具。数据集与代码已开源：https://github.com/anshulsc/MirageTVQA。
原文摘要 · Abstract (English)
The impressive performance of VLMs is largely measured on benchmarks that fail to capture the complexities of real-world scenarios. Existing datasets for tabular QA, such as WikiTableQuestions and FinQA, are overwhelmingly monolingual (English) and present tables in a digitally perfect, clean format. This creates a significant gap between research and practice. To address this, we present \textbf{MirageTVQA}, a new benchmark designed to evaluate VLMs on these exact dimensions. Featuring nearly 60,000 QA pairs across 24 languages, MirageTVQA challenges models with tables that are not only multilingual but also visually imperfect, incorporating realistic noise to mimic scanned documents. Our evaluation of the leading VLMs reveals two primary failure points: a severe degradation in performance (over 35\% drop for the best models) when faced with visual noise and a consistent English-first bias where reasoning abilities fail to transfer to other languages. MirageTVQA provides a benchmark for measuring and driving progress towards more robust VLM models for table reasoning. The dataset and the code are available at: https://github.com/anshulsc/MirageTVQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。