让视觉对象直接嵌入文本，提升模型对图像物体的精准理解。
MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
- 用视觉对象替换文本中的词语，实现局部跨模态对齐。
- 仅用5万样本效果超60万传统数据训练的模型。
- 适合需要精确物体定位与理解的多模态应用。
现有多模态大语言模型主要依赖图像-文本对进行预训练，将全局图像表征映射到长文本描述，但这种图像级对齐存在指代模糊问题：模型难以从全局表征中推断多个视觉对象与文本实体的对应关系，导致数据效率低且语义定位不准确。为此，我们提出多模态代码切换（MMCS），一种新型预训练范式，通过显式提供物体级监督来解决该问题。受语言学中代码切换现象启发，MMCS将视觉对象与文本交替嵌入，强制实现局部视觉-语言对齐。我们进一步构建了可扩展的数据合成流水线，生成包含77.3万条样本、具有精确物体-实体对应关系的预训练数据集。实验表明，MMCS具备极高数据效率：仅使用5万样本即可达到或超越使用60万图像-文本对训练的模型性能。此外，MMCS在不同模型规模下均显著提升视觉定位与感知能力。
原文摘要 · Abstract (English)
Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions. However, this image-level alignment suffers from referential ambiguity: models struggle to infer the correspondences between multiple visual objects and textual entities from the global representation, leading to data inefficiency and suboptimal semantic grounding. To address this, we propose MultiModal Code-Switching (MMCS), a novel pretraining paradigm that provides explicit object-level supervision. Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding. We further develop a scalable data synthesis pipeline to generate a pretraining dataset of 773K samples with accurate object-entity correspondences. Experiments show that MMCS is highly data-efficient: with only 50K samples, it matches or surpasses models trained on 600K image-text pairs. Furthermore, MMCS consistently improves visual grounding and perception capabilities across varying model scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。