arXiv:2410.01744cs.CVcs.CL2024-10被引 18

Leopard模型专精多图文文本理解,提升跨图推理能力。

Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks

  • 构建百万级高质量图文指令数据集,专注文本密集型多图任务。
  • 采用自适应高分辨率编码,动态分配视觉序列长度,平衡精度与效率。
  • 仅用120万样本即超越顶尖模型,开源可复现,适合多图文本分析场景。

文本密集型图像(如演示文稿、扫描文档、网页截图)在真实场景中广泛存在,涉及多张此类图像的任务尤为复杂,需理解单图内容并推理多图间的关联与逻辑流程。现有多模态大模型因两大挑战难以应对:一是缺乏高质量的文本密集型多图指令微调数据集;二是图像分辨率与视觉特征序列长度难以兼顾。为此,我们提出Leopard,一款专为多文本密集型图像任务设计的多模态大语言模型。首先,我们构建了约一百万条高质量的多模态指令微调数据,聚焦文本密集型多图场景。其次,提出自适应高分辨率多图编码模块,根据原始图像宽高比与分辨率动态优化视觉序列长度分配。在多个基准测试中,Leopard在文本密集型多图任务上持续优于Llama-3.2和Qwen2-VL等领先系统。尤为突出的是,仅使用120万训练实例即可实现卓越性能,所有数据与代码均开源,展现出高效率与有效性,相比依赖大规模私有数据的模型更具优势。代码与数据见https://github.com/tencent-ailab/Leopard。

原文摘要 · Abstract (English)

Text-rich images, where text serves as the central visual element guiding the overall understanding, are prevalent in real-world applications, such as presentation slides, scanned documents, and webpage snapshots. Tasks involving multiple text-rich images are especially challenging, as they require not only understanding the content of individual images but reasoning about inter-relationships and logical flows across multiple visual inputs. Despite the importance of these scenarios, current multimodal large language models (MLLMs) struggle to handle such tasks due to two key challenges: (1) the scarcity of high-quality instruction tuning datasets for text-rich multi-image scenarios, and (2) the difficulty in balancing image resolution with visual feature sequence length. To address these challenges, we propose Leopard, an MLLM tailored for handling vision-language tasks involving multiple text-rich images. First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios. Second, we proposed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length based on the original aspect ratios and resolutions of images. Experiments on a diverse set of benchmarks reveal that our model consistently outperforms state-of-the-art systems, such as Llama-3.2 and Qwen2-VL, in challenging text-rich, multi-image evaluations. Remarkably, our approach achieves outstanding performance using only 1.2M training instances, all of which are fully open-sourced, demonstrating both high efficiency and effectiveness compared to models trained on large-scale in-house data. Our code and data are available at https://github.com/tencent-ailab/Leopard.

多图理解文本密集多模态模型开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。