大权重是防止注意力模型失效的关键，而非跳跃连接。
Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse
- 提出层坍缩概念，揭示小权重致模型退化为单层
- 证明即使有跳跃连接，小权重仍导致表达力丧失
- 强调大权重对模型性能的决定性作用，适合架构研究者
注意力机制是现代大语言模型的核心。前向与反向计算通常需二次时间，此前工作表明：除非模型权重较小，否则二次时间无法避免；而小权重下可实现近线性时间。本文证明：大权重是防止一种称为层坍缩的表征能力丧失所必需的，即整个网络可被单层网络良好逼近。因此，表达性变换器的注意力计算时间复杂度不可避免地为二次。我们引入的层坍缩是[Dong, Cordonnier, and Loukas ICML 2021]提出的秩坍缩的变体。他们发现，在小权重且含跳跃连接的自注意力网络中，秩坍缩必然发生，通常被解释为跳连必要性的依据。但我们的结果表明，即便有跳连，若权重过小，层坍缩依然存在。因此，仅大权重能防止此类表征缺陷，而非跳连。
原文摘要 · Abstract (English)
Attention mechanisms lie at the heart of modern large language models (LLMs). Straightforward algorithms for forward and backward (gradient) computation take quadratic time, and a line of work initiated by [Alman and Song NeurIPS 2023] and [Alman and Song NeurIPS 2024] has shown that quadratic time is necessary unless the model weights are small, in which case almost linear time algorithms are possible. In this paper, we show that large weights are necessary to avoid a strong preclusion to representational strength we call layer collapse, which means that the entire network can be approximated well by a network with only a single layer. Thus, the quadratic running time of attention is unavoidable for expressive transformers. The notion of layer collapse that we introduce is a variant on the notion of rank collapse from the work of [Dong, Cordonnier, and Loukas ICML 2021]. They showed that in Self Attention Networks with small weights and with skip connections, rank collapse must occur. This is typically interpreted as justifying the necessity of skip connections in expressive networks. However, our result shows that even with skip connections, if the weights are small, then layer collapse still occurs. Thus, only large weights, and not skip connections, can prevent these representational weaknesses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。