发现注意力层隐含专家混合机制，缓解头坍缩问题
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
- 揭示普通与下沉注意力天然形成专家混合结构
- 提出感知下沉的训练方法，实现头负载均衡
- 适用于多种注意力架构，提升模型性能
大型语言模型常过度关注首词，即注意力坍缩现象。现有方法如GPT-OSS的Sink Attention和Qwen3-Next的Gated Attention虽有改善，但缺乏对各类注意力机制关系的系统分析。本文从理论与实证两方面证明，原始注意力与下沉注意力在注意力层中自然构成混合专家（MoE）机制，解释了先前观察到的头坍缩现象——仅少数注意力头参与生成。为缓解该问题，我们提出一种感知下沉的训练算法，引入辅助负载均衡损失。大量实验表明，该方法在原始注意力、下沉注意力及门控注意力中均实现了有效头负载均衡，并提升模型性能。本研究为注意力机制提供了新视角，鼓励进一步探索注意力层中的内在MoE结构。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these attention mechanisms is lacking. In this work, we provide both theoretical and empirical evidence demonstrating that the sink in Vanilla Attention and Sink Attention naturally construct a Mixture-of-Experts (MoE) mechanism within attention layers. This insight explains the head collapse phenomenon observed in prior work, where only a fixed subset of attention heads contributes to generation. To mitigate head collapse, we propose a sink-aware training algorithm with an auxiliary load balancing loss designed for attention layers. Extensive experiments show that our method achieves effective head load balancing and improves model performance across Vanilla Attention, Sink Attention, and Gated Attention. We hope this study offers a new perspective on attention mechanisms and encourages further exploration of the inherent MoE structure within attention layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。