arXiv:2606.08105cs.LG2026-06被引 1

同一注意力模式可能对应两种不同计算机制,需区分干预。

A Unifying View of Attention Sinks: Two Algorithms, Two Solutions

论文配图:A Unifying View of Attention Sinks: Two Algorithms, Two Solutions
图 1 · 摘自论文原文
  • 发现注意力汇聚存在自适应置零与全局广播两种本质机制。
  • 实验证明两类机制在模型中并存,且影响性能与稳定性。
  • 建议结合门控与寄存器策略,提升模型鲁棒性与效果。

当注意力集中于单一标记(即‘注意力汇点’)时,模型实际在执行何种计算?尽管软最大值变换器中此类现象普遍存在,但相似的视觉模式可能隐藏着根本不同的算法机制。本文揭示:视觉上相似的汇点模式可对应两种不同机制:(i) 自适应置零,即头通过路由至空标记来抑制更新;(ii) 广播机制,即汇点聚合并重新分发全局信息。两者均充当无有效计算时的安全终点。诸如门控或寄存器等干预手段之所以有效,是因为它们隐式针对其中一种机制——门控假设为置零,寄存器假设为广播。每种机制留下独特痕迹:置零汇点值范数极低,广播汇点导致低秩输出。我们在合成任务中形式化这些特征,并用于开发实用诊断工具。应用于预训练视觉变换器时发现,两类机制在大规模模型中均存在:汇点从早期层的分类标记(CLS)演变为深层的图像块,且集中在特定头部。令人惊讶的是,专为广播设计的寄存器标记也被用于置零功能,表明单一干预不足。结合门控与寄存器带来互补增益,在稳定性和性能上均有提升。总体而言,相同注意力模式可能反映截然不同的计算,有效干预必须首先明确模型真实计算逻辑。

原文摘要 · Abstract (English)

When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: {i} adaptive nop, where a head suppresses its update by routing to a null token, and {ii} broadcast, where a sink aggregates and redistributes global information. In that case, sinks serve an analogous role: a safe destination when there is nothing useful to compute. Proposed interventions like gating or registers work because they implicitly target one or the other, revealing a duality between method and assumed mechanism: gating implicitly assumes nop; registers implicitly assume broadcast. Each mechanism leaves distinct traces (nop sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs) which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers, and concentrate in specialized heads. Strikingly, register tokens, designed for broadcast, are repurposed to also serve nop, confirming that neither intervention alone suffices. Combining gating with registers yields complementary gains in stability and performance. Overall, we find that the same attention pattern can reflect two very different computations and effective intervention requires first asking what the model is actually computing.

注意力机制变换器模型诊断神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。