通过重建视觉标记提升大模型对图像的理解能力
ViCToR: Improving Visual Comprehension via Token Reconstruction for Pretraining LMMs
- 用可学习的视觉标记池和匈牙利算法选择相关标记替换原图标记
- 在300万图文数据上预训练，多项评测性能超越LLaVA-NeXT-8B
- 适合追求图像理解精度的多模态模型研究者使用
大型多模态模型（LMM）在预训练中常面临模态表示差异：语言嵌入稳定，而视觉表示易受上下文噪声（如背景杂乱）影响。为此，我们提出一种名为ViCToR（通过标记重建实现视觉理解）的新预训练框架。ViCToR采用可学习的视觉标记池，并利用匈牙利匹配算法从中选取语义相关的标记进行视觉标记替换。通过引入视觉标记重建损失与密集语义监督，使模型学习到保留高视觉细节的标记，从而增强大语言模型对视觉信息的理解。在300万公开可用的图像与标题上完成预训练后，ViCToR在MMStar、SEED$^I$和RealWorldQA基准上分别比LLaVA-NeXT-8B提升10.4%、3.2%和7.2%。代码已开源。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) often face a modality representation gap during pretraining: while language embeddings remain stable, visual representations are highly sensitive to contextual noise (e.g., background clutter). To address this issue, we introduce a visual comprehension stage, which we call ViCToR (Visual Comprehension via Token Reconstruction), a novel pretraining framework for LMMs. ViCToR employs a learnable visual token pool and utilizes the Hungarian matching algorithm to select semantically relevant tokens from this pool for visual token replacement. Furthermore, by integrating a visual token reconstruction loss with dense semantic supervision, ViCToR can learn tokens which retain high visual detail, thereby enhancing the large language model's (LLM's) understanding of visual information. After pretraining on 3 million publicly accessible images and captions, ViCToR achieves state-of-the-art results, improving over LLaVA-NeXT-8B by 10.4%, 3.2%, and 7.2% on the MMStar, SEED$^I$, and RealWorldQA benchmarks, respectively. Code is available at https://github.com/deepglint/Victor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。