arXiv:2608.12149cs.CL2026-08

发现混合注意力模型中激活峰值的两种新形态,揭示其演化规律。

Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

论文配图:Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
图 1 · 摘自论文原文
  • 首次系统研究混合线性注意力大模型的巨量激活,发现前置尖峰与层间平台两种形态。
  • 随着全注意力层密度增加,前置尖峰通过平台持续连接,最终恢复稳定激活模式。
  • 适用于研究模型内部动态、优化注意力机制或设计高效大模型的学者。

我们对层交错式混合线性注意力大语言模型中的巨量激活(MAs)进行了首次系统研究,揭示了两种与架构对齐的形态:巨量激活在全注意力层前持续出现前置尖峰(PAS),并可在中间的线性注意力层中持续存在,形成层间平台(ISP)。随着全注意力层变得更密集,连续的前置尖峰通过平台逐步连接,最终恢复全注意力模型所具有的稳定巨量激活形态。该组织结构在五种线性注意力架构、六种混合配置、五个数据领域及覆盖1.2B至397B参数的代表性开源混合模型中均呈现。在高达1.3B规模的基于GDN的混合模型控制预训练中,两类形态早期即出现,且对输出门控响应不对称:全注意力输出门控显著抑制其绝对幅值但不破坏层间结构;移除GDN门控则仅带来较小增幅。机制分析表明,二者共享一个由激活抵消时机决定的生命周期:前置尖峰遵循局部写入-消解过程,而平台的持久性符合延迟抵消特征。在全注意力极限下,该机制恢复全注意力模型典型的稳定巨量激活形态。代码已公开于 https://github.com/StartLuxLabs/Massive-Activations-HLA。

原文摘要 · Abstract (English)

We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartLuxLabs/Massive-Activations-HLA.

大模型注意力机制激活分析神经科学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。