发现大模型深层注意力头会'丢弃'冗余信息，据此可高效压缩模型。
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
- 通过分析注意力头的BOS汇流程度，识别出冗余组件。
- 在多个模型上剪除高汇流头后性能几乎不变，剪枝更精准。
- 适合做模型轻量化与结构优化的研究者参考。
大型语言模型存在显著冗余，但为何高层层中某些组件更冗余仍缺乏系统解释。本文揭示了BOS汇流现象是导致层间敏感性的关键机制：高BOS汇流得分的注意力头与功能冗余强相关，尤其在深层中贡献极小，仅作为多余注意力权重的‘垃圾堆’。这为先前研究报道的结构性冗余提供了具体的功能解释。基于此，我们提出一种简单剪枝策略，移除高BOS汇流头。在Gemma-3、Llama-3.1和Qwen3上的实验表明，该方法比基于权重或激活的准则更可靠地识别冗余组件，在激进剪枝下仍能保持接近稠密基线的性能。此外，汇流头行为在不同序列长度下保持稳定。结果表明，注意力结构特性比基于幅度的方法更直观、更稳健，适合作为模型压缩的基础。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in higher layers, are more redundant has remained elusive. In this work, we identify the BOS sink phenomenon as a key mechanism driving this layer-wise sensitivity. We show that attention heads with high BOS sink scores are strongly associated with functional redundancy: such heads, especially in deeper layers, contribute little to predictive performance and effectively serve as \emph{dumping grounds} for superfluous attention weights. This provides a concrete functional explanation for the structural redundancy reported in prior studies. Leveraging this insight, we introduce a simple pruning strategy that removes high-BOS sink heads. Experiments on Gemma-3, Llama-3.1, and Qwen3 demonstrate that this approach identifies redundant transformer components more reliably than weight- or activation-based criteria, while preserving performance close to dense baselines even under aggressive pruning. Moreover, we find that the behavior of sink heads remains stable across different sequence lengths. Overall, our results suggest that structural properties of attention offer a more intuitive and robust basis for model compression than magnitude-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。