通过掩码生成提升文档理解中的图文对齐,减少幻觉。
Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding
- 设计VQA+掩码生成联合任务,同时实现语义与空间对齐。
- 在600万数据集上训练,8B模型在文档任务中表现领先。
- 适合需要精准图文对齐的文档分析场景,如票据、表单处理。
多模态大语言模型(MLLM)为文档理解带来了新维度,使语言模型具备视觉理解能力;然而,如何设计合适的图像-文本预训练任务以连接文档级MLLM中的视觉与语言模态仍缺乏探索。本文提出一种新型视觉-语言对齐方法,将核心问题建模为视觉问答与掩码生成(VQAMask)任务,同时优化两项任务:基于VQA的文本解析和掩码生成。前者使模型在语义层面隐式对齐图像与文本,后者引入额外掩码生成器(推理时丢弃),在空间感知层面显式保证图像内文字与其对应区域的一致性。两者协同可有效防止文本解析时的模型幻觉,并促进空间感知特征表示学习。为支持该任务,我们构建了大规模图像-掩码生成流水线,创建包含600万样本的大型数据集MTMask6M。实验表明,引入掩码生成任务可取得具有竞争力的文档级理解性能。基于此,我们提出Marten,一种面向文档理解的高效训练多模态大模型。大量实验显示,Marten在8B-MLLM中于文档中心任务上持续取得显著提升。代码与数据集见https://github.com/PriNing/Marten。
原文摘要 · Abstract (English)
Multi-modal Large Language Models (MLLMs) have introduced a novel dimension to document understanding, i.e., they endow large language models with visual comprehension capabilities; however, how to design a suitable image-text pre-training task for bridging the visual and language modality in document-level MLLMs remains underexplored. In this study, we introduce a novel visual-language alignment method that casts the key issue as a Visual Question Answering with Mask generation (VQAMask) task, optimizing two tasks simultaneously: VQA-based text parsing and mask generation. The former allows the model to implicitly align images and text at the semantic level. The latter introduces an additional mask generator (discarded during inference) to explicitly ensure alignment between visual texts within images and their corresponding image regions at a spatially-aware level. Together, they can prevent model hallucinations when parsing visual text and effectively promote spatially-aware feature representation learning. To support the proposed VQAMask task, we construct a comprehensive image-mask generation pipeline and provide a large-scale dataset with 6M data (MTMask6M). Subsequently, we demonstrate that introducing the proposed mask generation task yields competitive document-level understanding performance. Leveraging the proposed VQAMask, we introduce Marten, a training-efficient MLLM tailored for document-level understanding. Extensive experiments show that our Marten consistently achieves significant improvements among 8B-MLLMs in document-centric tasks. Code and datasets are available at https://github.com/PriNing/Marten.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。