arXiv:2506.16691cs.CV2025-06被引 5

通过内部特征调制,让视觉语言模型更高效融合多模态信息。

LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation

  • 在语言模型内部动态调整参数,直接融合视觉信息
  • 相比现有模型减少94%计算量,推理速度提升3.1倍
  • 适合需要实时多模态推理的轻量化部署场景

尽管大型视觉语言模型(LVLMs)取得显著进展,现有方法仍面临视觉-语言融合效率低下的根本瓶颈:要么破坏模型固有结构,要么引入严重长上下文计算负担,严重限制可扩展性与效率。本文重新思考多模态融合,提出LaVi,一种通过大语言模型(LLMs)内部特征调制实现无缝高效视觉-语言融合的新颖LVLM。不同于主流基于视觉令牌拼接的方法,LaVi通过引入轻量且自适应的变换机制,将视觉上下文以逐标记的视觉条件差分形式注入层归一化仿射参数中,从而直接基于视觉输入调制语言隐藏状态,实现精确的多模态对齐,同时保留LLM的语言先验,并大幅降低计算成本。在15个图像与视频基准上的广泛评估表明,LaVi不仅达到最先进的多模态性能,还显著提升效率:相较于LLaVA-OV-7B,LaVi降低94.0%的浮点运算量(FLOPs),推理速度提升3.1倍,内存占用减半,确立了其在实时多模态推理中的可扩展性与实用性。代码与模型将很快发布。

原文摘要 · Abstract (English)

Despite the impressive advancements of Large Vision-Language Models (LVLMs), existing approaches suffer from a fundamental bottleneck: inefficient visual-language integration. Current methods either disrupt the model's inherent structure or introduce severe long-context computational burden, severely limiting scalability and efficiency. In this paper, we rethink multimodal integration and present LaVi, a novel LVLM that enables seamless and efficient vision-language fusion through internal feature modulation within the Large Language Models (LLMs). Unlike dominant LVLMs that rely on visual token concatenation, LaVi bypasses long-context expansion by introducing a lightweight and adaptive transformation, which incorporates visual context by injecting token-wise vision-conditioned deltas into the affine parameters of layer normalization. This mechanism directly modulates linguistic hidden states based on visual input, ensuring precise vision-language alignment while preserving the LLM's linguistic priors and drastically reducing computational costs. Extensive evaluations across 15 image and video benchmarks demonstrate that LaVi not only achieves state-of-the-art multimodal performance but also dramatically enhances efficiency. Compared to LLaVA-OV-7B, LaVi reduces FLOPs by 94.0%, improves inference speed by 3.1 times, and cuts memory usage in half - establishing LaVi as a scalable and practical solution for real-time multimodal reasoning. The code and models will be released soon.

视觉语言模型高效推理特征调制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。