arXiv:2604.03316cs.CV2026-04

提出统一框架解析视觉语言模型中的注意力汇聚点,平衡全局与局部感知。

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

论文配图:When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
图 1 · 摘自论文原文
  • 区分视觉编码器和语言模型产生的两类注意力汇聚点
  • 发现汇聚点虽增强全局理解但会压制细粒度视觉信息
  • 设计轻量级模块动态调节汇聚点权重,适配多任务场景

注意力汇聚点(attention sinks)指吸引异常多注意力的标记。尽管在单模态变换器中已有研究,其在大型视觉-语言模型(LVLM)中的跨模态影响仍不明确:它们是冗余产物还是必要全局先验?本文首次将视觉汇聚点分为两类:源自视觉变压器(V-sinks)和来自深层语言模型(L-sinks)。基于新定义,分析揭示根本性性能权衡:汇聚点虽有效编码全局场景先验,但其主导性会抑制局部感知所需的细粒度视觉证据。进一步识别出特定功能层,调节这些汇聚点对下游性能影响最大。为此,提出分层汇聚门控(LSG),一种轻量、即插即用的模块,可动态调节V-sink与其他视觉标记的注意力贡献。LSG通过标准的下一个词预测训练,无需任务特定监督,且保持LVLM主干冻结。在多数层中,LSG在代表性多模态基准上取得提升,有效平衡全局推理与精确局部证据。

原文摘要 · Abstract (English)

Attention sinks are defined as tokens that attract disproportionate attention. While these have been studied in single modality transformers, their cross-modal impact in Large Vision-Language Models (LVLM) remains largely unexplored: are they redundant artifacts or essential global priors? This paper first categorizes visual sinks into two distinct categories: ViT-emerged sinks (V-sinks), which propagate from the vision encoder, and LLM-emerged sinks (L-sinks), which arise within deep LLM layers. Based on the new definition, our analysis reveals a fundamental performance trade-off: while sinks effectively encode global scene-level priors, their dominance can suppress the fine-grained visual evidence required for local perception. Furthermore, we identify specific functional layers where modulating these sinks most significantly impacts downstream performance. To leverage these insights, we propose Layer-wise Sink Gating (LSG), a lightweight, plug-and-play module that dynamically scales the attention contributions of V-sink and the rest visual tokens. LSG is trained via standard next-token prediction, requiring no task-specific supervision while keeping the LVLM backbone frozen. In most layers, LSG yields improvements on representative multimodal benchmarks, effectively balancing global reasoning and precise local evidence.

视觉语言模型注意力机制多模态模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。