揭示多头注意力压缩中旋转编码的应用方式如何影响模型容量
A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention
- 用随机矩阵理论分析注意力权重矩阵谱分布
- 发现压缩引发局部容量瓶颈和秩坍缩现象
- 解耦旋转编码可避免谱碎片化，适合高效压缩场景
本文研究多头潜在注意力（MLA）这一常用内存压缩策略对Transformer模型预训练期间内部容量的影响。通过轻量级的Marchenko-Pastur诊断方法，分析训练过程中$W_{Q}W_{K}^ op$ Gram矩阵的谱特性，对比三种变体：标准多头注意力（MHA）基线、压缩前应用旋转编码的MLA-PreRoPE，以及所有头共享单一旋转子向量的MLA-Decoupled。随机矩阵分析揭示三个关键发现：一）容量瓶颈在特定层早期出现并持续传播，破坏整体与异常方向的平衡；二）这些尖峰与秩坍缩同步，使模型表达力集中于狭窄子空间；三）仅解耦变体能阻止此连锁反应，维持广泛的谱支持并抑制异常值生成。结果表明，旋转编码的应用方式与压缩位置同等重要。跨头共享旋转组件可缓解谱碎片化，保持表征容量。
原文摘要 · Abstract (English)
In this work, we study how multi-head latent attention (MLA), a popular strategy for compressing key/value memory, affects a transformer's internal capacity during pretraining. Using a lightweight suite of Marchenko-Pastur (MP) diagnostics, we analyze the spectrum of the $W_{Q}W_{K}^\top$ gram matrix throughout training, comparing three variants: the standard multi-head attention (MHA) baseline, MLA-PreRoPE with rotary applied before compression, and MLA-Decoupled, which shares a single rotary sub-vector across all heads. Our random matrix analysis reveals \textbf{three key findings:} \textbf{ i)} capacity bottlenecks emerge locally: both MHA and MLA-PreRoPE exhibit sharp, early spikes in specific layers that persist and propagate, disrupting the balance between bulk and outlier directions; \textbf{ ii)} these spikes coincide with rank collapse, concentrating the model's expressivity into narrow subspaces; \textbf{ iii)} only the decoupled variant prevents this cascade, maintaining broad spectral support and suppressing outlier formation across layers. These results underscore that \emph{how} rotary embeddings are applied is just as critical as \emph{where} compression occurs. Sharing rotary components across heads mitigates spectral fragmentation and preserves representational capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。