揭示多头潜在注意力如何分离内容与位置信息
Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
- 通过共享低秩瓶颈实现内容与位置解耦
- 内容信息保留率达98%，位置信息被有效丢弃
- 发现语义核心层与集中式归纳头结构
多头潜在注意力（MLA）在DeepSeek-V2中通过共享的低秩瓶颈（cKV）压缩键值对，在推理时实现81%的KV缓存减少。尽管已广泛用于大规模生产模型，此前尚无研究探讨该瓶颈保留或丢弃的信息，以及其对内部Transformer电路的影响。本文首次对MLA进行系统性机制可解释性分析，训练了一个114M参数的Transformer（在网页/代码/数学混合数据上预训练，微调于TinyStories），并结合SVD、注意力头分类、线性探测及破坏-归因分析进行研究。主要发现包括：（1）cKV瓶颈学习到纯内容表示，保留实体身份（98%保留率），同时丢弃位置信息，验证了通过RoPE实现内容与位置的解耦；（2）归纳头集中在单一层次（第12层），不同于标准MHA中的分布形态；（3）第15层为“语义枢纽”层，具有最高SVD有效秩和最强破坏-归因分数；（4）瓶颈整体过配置，平均仅使用46%容量。这些结果表明MLA并非被动压缩，而是主动重塑模型的内容组织、位置处理与电路结构。本研究为理解其机制提供初步数据点，具体局限见第5节。
原文摘要 · Abstract (English)
Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference. Despite its adoption in massive production models, no prior work has studied what information this bottleneck preserves or discards, nor how it reshapes internal transformer circuits. We present the first comprehensive mechanistic interpretability study of MLA, training a 114M-parameter transformer (pretrained on a web/code/math mixture, fine-tuned on TinyStories) and analyzing its representations through SVD, attention head taxonomy, linear probing, and a disruption-attribution analysis. Our key findings are: (1) the cKV bottleneck learns a pure content representation, preserving entity identity (98% retention) while discarding positional information, validating MLA's separation of content from position via RoPE; (2) induction heads co-locate at a single layer (Layer 12), unlike their distributed formation in standard MHA; (3) a single "semantic hub" layer (Layer 15) simultaneously exhibits the highest SVD effective rank and strongest disruption-attribution score; and (4) the bottleneck is globally over-provisioned, using only 46% of its capacity on average. These findings suggest MLA does not merely compress attention passively, but reshapes how the model organizes content, position, and circuit structure. We view this as an initial data point and detail scope limitations in Section 5.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。