让大模型同时当语言和视觉编码器，提升多模态理解效果
Rethinking Visual Information Processing in Multimodal LLMs
- 用大模型自身做视觉编码，通过三处改进实现跨模态融合
- 在多个评测上超越原版LLaVA，甚至超过参数量翻倍的模型
- 适合研究多模态大模型融合机制或想优化视觉理解的开发者
尽管LLaVA架构在多模态任务中表现卓越，但其设计存在文本与视觉模态本质不匹配的问题，导致视觉特征难以有效融合。本文提出一种新思路：将大语言模型（LLM）不仅作为语言模型，更视为强大的视觉编码器。为此，我们提出LLaViT——大语言模型扩展为视觉变换器，通过三项关键改进实现这一目标：(1) 为视觉模态学习独立的QKV投影；(2) 允许视觉令牌间的双向注意力；(3) 融合全局与局部视觉表征。在多种大模型上的大量受控实验表明，LLaViT在多个基准测试中显著优于基线方法LLaVA，甚至超越参数量翻倍的模型，证明了该方法在多模态建模中的有效性。
原文摘要 · Abstract (English)
Despite the remarkable success of the LLaVA architecture for vision-language tasks, its design inherently struggles to effectively integrate visual features due to the inherent mismatch between text and vision modalities. We tackle this issue from a novel perspective in which the LLM not only serves as a language model but also a powerful vision encoder. To this end, we present LLaViT - Large Language Models as extended Vision Transformers - which enables the LLM to simultaneously function as a vision encoder through three key modifications: (1) learning separate QKV projections for vision modality, (2) enabling bidirectional attention on visual tokens, and (3) incorporating both global and local visual representations. Through extensive controlled experiments on a wide range of LLMs, we demonstrate that LLaViT significantly outperforms the baseline LLaVA method on a multitude of benchmarks, even surpassing models with double its parameter count, establishing a more effective approach to vision-language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。