arXiv:2606.20743cs.LGcs.AI2026-06

拆分模型计算与输出通道,发现大激活仍会自动生成,说明其非设计缺陷而是功能需要。

Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test

论文配图:Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test
图 1 · 摘自论文原文
  • 将残差流分为可写临时区和只读保护区,测试大激活是否因功能重叠产生
  • 在160M和290M模型中,大激活仍会在保护区重现,且更集中于序列起始位置
  • 强稀疏惩罚反而让激活更持久、更集中,证明其具备架构鲁棒性

训练后的Transformer模型会稳定出现大规模激活,即少数隐藏维度的值远超中位数,且集中在序列起始标记上。这种现象是残差流同时承担计算临时存储与输出功能所导致的可移除副作用,还是功能必需的内在机制,尚存争议。本文通过架构干预直接检验该假设:提出Ledger Residuals架构,将残差流分为可自由覆盖的临时计算区(Deliberation)和仅用于解码的受保护累积区(Commitment)。若大激活仅因双重角色导致,则专用输出通道应能消除其需求。结果发现:在160M与290M规模的匹配损失语言模型中,模型仍会在保护区内重建典型的固定维度、起始标记处的大激活。重建特征幅值虽小于标准Transformer,但更聚焦于起始标记;更强的稀疏惩罚使其更持久、更集中,而非消失。因此,大激活表现出架构鲁棒性——只要模型从某表示中解码,它们就会重新出现,这符合其功能性而非偶然性的预期。代码与架构已公开。

原文摘要 · Abstract (English)

Trained transformers reliably develop massive activations, a small number of hidden dimensions whose magnitude is far above the median and which concentrate on the sequence-start token. Whether these outliers are a removable artifact of the residual stream's overloaded read and write role, or instead a functional necessity, is actively debated. We test the artifact hypothesis directly, with an architectural intervention. Our architecture, Ledger Residuals, splits the residual stream into a mutable scratch stream (Deliberation) that intermediate computation may freely overwrite and a protected, decode-only accumulator (Commitment) that holds the representation the model reads out. If massive activations exist only because one stream is forced to be both scratchpad and answer, then a dedicated answer channel should remove the need for them. We find that it does not. In matched-loss language models at the 160M and 290M scales, the model rebuilds the canonical fixed-dimension, start-token outlier inside the protected channel. The rebuilt feature is smaller in magnitude than in a standard transformer but more sharply concentrated on the start token, and a stronger sparsity penalty makes it more persistent and more concentrated still, rather than removing it. Massive activations therefore look architecturally robust: they re-emerge in whichever representation the model decodes from, which is what we would expect if they are functional rather than incidental. We release our architecture and measurement code.

Transformer残差流大激活架构分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。