评测多语言图文理解模型在跨语言场景下的表现并提出改进方法
Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective
- 构建跨语言图文问答基准XT-VQA，涵盖五大数据集与新收集的XPaperQA
- 发现主流模型在跨语言任务中性能大幅下降，最高降幅达37.6%
- 提出基于互信息优化的MVCL-MI方法，有效缩小跨语言性能差距
近年来大型视觉语言模型（LVLMs）在图表、表格和文档等富文本图像上展现出出色的推理能力。然而，图像中的丰富文本可能使模型对语言更敏感，从而在图像文本与指令语言不一致的跨语言场景下表现不佳。为此，本文提出XT-VQA（跨语言富文本视觉问答）基准，用于评估LVLM在跨语言输入下的表现。该基准整合了五个现有富文本VQA数据集及新构建的XPaperQA数据集，覆盖多种需忠实识别与理解视觉信息的场景。对主流LVLM在XT-VQA上的评估显示，即使具备多语言能力的模型，在跨语言场景中性能也显著下降，平均降幅达37.6%。互信息分析表明，跨语言问题未能充分激活相关视觉信息。为此，本文提出MVCL-MI（最大化视觉-语言跨语言互信息）方法，通过最小化单语与跨语言输出逻辑值之间的KL散度，实现从单语到跨语言的知识蒸馏，建立视觉-文本跨语言对齐。实验结果表明，该方法有效缓解了跨语言性能差距，同时保留了模型原有能力。
原文摘要 · Abstract (English)
Recent Large Vision-Language Models (LVLMs) have shown promising reasoning capabilities on text-rich images from charts, tables, and documents. However, the abundant text within such images may increase the model's sensitivity to language. This raises the need to evaluate LVLM performance on cross-lingual text-rich visual inputs, where the language in the image differs from the language of the instructions. To address this, we introduce XT-VQA (Cross-Lingual Text-Rich Visual Question Answering), a benchmark designed to assess how LVLMs handle language inconsistency between image text and questions. XT-VQA integrates five existing text-rich VQA datasets and a newly collected dataset, XPaperQA, covering diverse scenarios that require faithful recognition and comprehension of visual information despite language inconsistency. Our evaluation of prominent LVLMs on XT-VQA reveals a significant drop in performance for cross-lingual scenarios, even for models with multilingual capabilities. A mutual information analysis suggests that this performance gap stems from cross-lingual questions failing to adequately activate relevant visual information. To mitigate this issue, we propose MVCL-MI (Maximization of Vision-Language Cross-Lingual Mutual Information), where a visual-text cross-lingual alignment is built by maximizing mutual information between the model's outputs and visual information. This is achieved by distilling knowledge from monolingual to cross-lingual settings through KL divergence minimization, where monolingual output logits serve as a teacher. Experimental results on the XT-VQA demonstrate that MVCL-MI effectively reduces the visual-text cross-lingual performance disparity while preserving the inherent capabilities of LVLMs, shedding new light on the potential practice for improving LVLMs. Codes are available at: https://github.com/Stardust-y/XTVQA.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。