通过分层视觉注入机制，10倍降低视觉语言模型计算量。
HiMix: Reducing Computational Complexity in Large Vision-Language Models
- 仅让语言序列全传播，视觉信息在特定层与语言交互
- 在多模型上实现语言解码器计算成本降低10倍
- 适合需要高效推理的视觉语言应用开发者
得益于大语言模型和模态对齐技术的进步，现有大型视觉-语言模型（LVLM）在多种场景中取得了显著性能。然而，过高的计算复杂度限制了其在实际应用中的普及。我们指出，计算复杂度的主要瓶颈在于模型计算中引入了冗余的视觉序列。这一发现源于对语言解码器中视觉与语言信息传输效率的重新评估。为此，我们提出一种新型分层视觉-语言交互机制——分层视觉注入混合注意力（HiMix）。在HiMix中，仅语言序列进行完整前向传播，而视觉序列仅在每个语言解码器层的特定阶段与语言交互。令人瞩目的是，该方法在几乎不损失性能的情况下显著降低计算复杂度：在多个LVLM模型上，语言解码器的计算成本降低了10倍。这凸显了本方法的优势，也期望为视觉-语言理解领域带来新视角。
原文摘要 · Abstract (English)
Benefiting from recent advancements in large language models and modality alignment techniques, existing Large Vision-Language Models(LVLMs) have achieved prominent performance across a wide range of scenarios. However, the excessive computational complexity limits the widespread use of these models in practical applications. We argue that one main bottleneck in computational complexity is caused by the involvement of redundant vision sequences in model computation. This is inspired by a reassessment of the efficiency of vision and language information transmission in the language decoder of LVLMs. Then, we propose a novel hierarchical vision-language interaction mechanism called Hierarchical Vision injection for Mixture Attention (HiMix). In HiMix, only the language sequence undergoes full forward propagation, while the vision sequence interacts with the language at specific stages within each language decoder layer. It is striking that our approach significantly reduces computational complexity with minimal performance loss. Specifically, HiMix achieves a 10x reduction in the computational cost of the language decoder across multiple LVLM models while maintaining comparable performance. This highlights the advantages of our method, and we hope our research brings new perspectives to the field of vision-language understanding. Project Page: https://xuange923.github.io/HiMix
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。