arXiv:2605.30265cs.CVcs.CL2026-05

让视觉语言模型学会跨模态等价表示,提升推理鲁棒性。

LoMo: Local Modality Substitution for Deeper Vision-Language Fusion

论文配图:LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
图 1 · 摘自论文原文
  • 通过动态替换文本为图像构建混合模态序列,增强跨模态对齐。
  • 在13个基准上平均提升2.67~2.82点,显著改善推理性能。
  • 适用于各类基础模型,尤其适合需强跨模态融合的场景。

视觉语言模型(VLMs)在众多理解与推理任务中取得显著进展,主要依赖大规模图文训练实现多模态融合。理想情况下,将文本问题替换为其渲染图像应不影响模型表现。但实践中,模态替换导致性能大幅下降。我们将其归因于现有训练数据中的固有偏见:在图像描述、VQA、OCR及网络爬取的交错数据集中,文本通常作为语言查询,图像则作为视觉参考,角色不对称。这种数据偏差使模型在不同模态间获取信息时表现出明显偏好,无法对齐语义等价内容的跨模态表示,导致模态替换下推理脆弱。为此,我们提出局部模态替换(LoMo),一种轻量级、与架构无关的数据筛选范式,旨在提供跨模态表示不变性的监督。LoMo通过将单模态提示重构为无缝交织的多模态序列,动态选择目标文本片段并重构成渲染图像,从而保持“文本-视觉-文本”载体间的语义一致性。在13个多样化多模态基准上的广泛实验表明,LoMo显著提升了整体多模态推理能力,实现了更深层次的跨模态融合。具体而言,其在基础模型上均表现优异,相较标准SFT分别在LLaVA-OneVision-1.5-8B和Qwen3.5-9B上提升2.67点和2.82点。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have achieved substantial progress across a wide range of understanding and reasoning tasks, driven by large-scale image-text training aimed at multimodal fusion. Ideally, replacing a textual question with its rendered-image counterpart should leave model performance essentially unaffected. In practice, however, such modality substitution induces dramatic performance degradation. We attribute this "carrier sensitivity" issue to an inherent bias in current training corpora. Across prevalent datasets such as image captioning, VQA, OCR, and web-sourced interleaved data, text and images are typically organized into distinct and asymmetric roles, with text serving as linguistic queries and images as visual references. Such data bias leads VLMs to exhibit distinct preferences for information acquisition across different modalities. Consequently, VLMs fail to align representations of semantically equivalent content across textual and visual carriers, making model reasoning fragile under modality substitution. To address this, we propose Local Modality Substitution (LoMo), a lightweight, architecture-agnostic data curation paradigm designed to provide supervision for cross-modal representational invariance between semantically equivalent text and image carriers. LoMo achieves this by reformulating single-modality prompts into seamlessly interleaved multimodal sequences. It dynamically selects target text spans and recasts them as rendered images, thereby preserving the same semantics across "text, visual, text" carriers. Extensive experiments across 13 diverse multimodal benchmarks demonstrate that LoMo significantly improves overall multimodal reasoning and yields deeper cross-modal fusion. Specifically, it delivers consistent gains across foundational models, improving over standard SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B.

视觉语言模型跨模态融合数据增强推理鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。