让视觉语言模型更懂图，动态打通多层视觉信息
From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion
- 构建动态多对多连接，让语言模型可选调用多层视觉特征
- 在18个评测上显著提升性能，最高增益达6.2个百分点
- 轻量高效，适配主流视觉语言模型，提升理解深度
视觉语言模型（VLMs）因采用单一、静态的异步连接方式，仅将视觉编码器输出传递给大语言模型（LLM），导致严重的视觉特征瓶颈。这种架构限制了LLM对分层视觉知识的全面对齐，影响其整合局部细节与全局语义的能力。为此，我们提出跨层注入（CLI）框架，构建视觉与语言模态间的动态多对多连接。CLI包含两个参数高效组件：自适应多投影（AMP）模块，用于融合不同层次的视觉特征；自适应门控融合（AGF）机制，使LLM能根据实时解码上下文选择性注入最相关视觉信息。我们将CLI集成至LLaVA-OneVision和LLaVA-1.5，在18个多样化基准上进行实验，结果表明性能显著提升，验证了CLI作为可扩展范式，能通过按需访问完整视觉层次，实现更深层次的多模态理解。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). This static architecture fundamentally limits the ability of LLMs to achieve comprehensive alignment with hierarchical visual knowledge, compromising their capacity to accurately integrate local details with global semantics into coherent reasoning. To resolve this, we introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an Adaptive Multi-Projection (AMP) module that harmonizes features from diverse vision layers, and an Adaptive Gating Fusion (AGF) mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate the effectiveness and versatility of CLI by integrating it into LLaVA-OneVision and LLaVA-1.5. Extensive experiments on 18 diverse benchmarks demonstrate significant performance improvements, establishing CLI as a scalable paradigm that unlocks deeper multimodal understanding by granting LLMs on-demand access to the full visual hierarchy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。