让多模态大模型更懂图像，解决视觉信息被忽视的问题
Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
- 在语言模型的联合语义空间中做图像掩码重建，直接增强视觉表征
- 在多个基准上显著提升视觉任务表现，尤其在密集视觉理解场景
- 适合关注视觉感知与多模态对齐的研究者或应用开发者
多模态大语言模型（MLLMs）在多模态任务中表现出色，但存在模态失衡问题：深层中视觉信息常被文本表示压制，导致视觉性能下降或幻觉。这源于训练主要依赖下一个文本词预测，缺乏直接的视觉监督信号，致使视觉表征逐层趋同。为此，我们提出潜空间视觉重建（LaVer），通过在语言模型的联合潜语义空间中进行掩码图像建模，使MLLMs学习更具区分性的视觉表示。该方法为模型提供直接视觉激活，显示其视觉注意力分配显著增加，表明视觉信息利用增强。跨多种基准的大量实验验证了该方法在不同场景下的优越性，尤其在需要密集视觉理解的任务中表现突出。LaVer代码已开源：https://github.com/Fir-lat/LaVer。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in multimodal tasks. Despite their impressive performance, MLLMs suffer from the modality imbalance issue, where visual information is often underutilized compared to textual representations in deeper layers, leading to degraded visual performance or hallucinations. This issue stems from the predominant reliance on next-text-token-prediction during training, which fails to provide direct visual supervisory signals, resulting in progressive homogenization of visual representations throughout the layers. To this end, we propose Latent Visual Reconstruction (LaVer), a novel training framework that facilitates MLLMs in learning more discriminative visual representations via masked image modeling in the joint latent semantic space of LLM. Our method offers direct visual activation to MLLMs, which exhibit increased visual attention allocation, indicating enhanced utilization of visual information. Extensive experiments across diverse benchmarks prove the superiority of our approach in various scenarios, especially those requiring dense visual capabilities. Code of LaVer is available at https://github.com/Fir-lat/LaVer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。