通过挖掘视觉语言模型的注意力稀疏性，实现多图推理加速。
BlindSight: Harnessing Sparsity for Efficient Vision-Language Models
- 利用图像间无注意力的规律，构建无需运行时开销的稀疏掩码。
- 在36K至300K提示长度下，注意力计算速度提升1.8至3.2倍。
- 适用于多种模型，精度损失仅0.78%，适合高效多图理解场景。
大型视觉语言模型（VLM）能联合处理文本与图像，但引入视觉数据会显著增加提示长度，导致首次输出时间（TTFT）变长。通过分析多图输入时的注意力模式，我们发现大量层中存在图像间无注意力现象。基于此，提出BlindSight：一种利用输入-模板感知的注意力稀疏掩码优化多图像VLM推理的方法，无运行时开销。我们通过数据集建立注意力头的无提示分类：密集型、汇聚型、图像内型及图像内+汇聚型。开发基于Triton的GPU内核以利用该稀疏性。BlindSight在提示长度36K至300K时，使注意力计算提速1.8至3.2倍。其在Qwen2-VL、Qwen2.5-VL和Gemma 3等模型上均表现良好，多图像理解基准平均精度仅下降0.78%。最后，我们倡导设计融合盲视启发的稀疏与密集层的高效VLM。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) enable joint processing of text and images. However, incorporating vision data significantly increases the prompt length, resulting in a longer time to first token (TTFT). This bottleneck can be alleviated by leveraging the inherent sparsity in the attention computation. Analyzing these attention patterns in VLMs when processing a series of images, we observe the absence of inter-image attention in a substantial portion of layers. Based on this, we propose BlindSight: an approach to optimize multi-image VLM inference using an input-template-aware attention sparsity mask with no runtime overhead. We utilize a dataset to derive a prompt-agnostic categorization for attention heads: Dense, Sink, Intra-Image, and Intra-Image+Sink. We develop a Triton-based GPU kernel to leverage this sparsity. BlindSight achieves a 1.8-3.2x speedup in the attention computation (prompt length 36K-300K). BlindSight generalizes across VLMs (Qwen2-VL, Qwen2.5-VL, Gemma 3), with only a 0.78% absolute accuracy degradation on average on multi-image comprehension benchmarks. Finally, we advocate for the design of efficient VLMs that combine BlindSight-inspired sparse and dense layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。