删掉注意力里的查询权重，模型性能不变还更省参数。
Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
- 用恒等矩阵替代查询权重，减少25%参数量
- 小模型实测性能持平，重分配参数后表现更优
- 适合追求高效部署的NLP研究者
本文从理论上分析了编码器和解码器仅用的Transformer中，查询、键、值权重三元组是否可简化。在温和假设下，证明其中任一权重可被单位矩阵替代，使注意力参数减少25%。若替换查询或键权重，优化过程也更简单：注意力得分仅依赖单一学习矩阵而非两矩阵乘积。在从头训练的小型GPT风格解码器模型上验证查询权重移除，发现简化模型在参数更少情况下性能与基线相当，且在重分配参数时表现更优。分析还揭示了一个结构性表达能力边界：在可解析的ReLU设定中，跳跃连接使MLP在固定宽度下进入一类通用不相交函数。这些发现为跨模态和大规模场景下的效率优化提供了新方向。
原文摘要 · Abstract (English)
We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that one of the Query, Key or Value weights are redundant and can be replaced with the identity matrix, reducing attention parameters by 25\%. If applied to the Query or Key weights, this also simplifies optimization: attention logits depend on a single learned weight matrix rather than on a product of two. Validating the Query weight removal on decoder-only GPT-style small models trained from scratch, we find that reduced models match baseline performance despite fewer parameters, and outperform baselines when saved parameters are reallocated. Our analysis has also led us to a structural expressivity boundary: in the mathematically tractable ReLU setting, skip connections push MLPs into a generically disjoint function class at fixed width. These findings motivate investigation across modalities and at scale, where the observed stability and efficiency gains may prove most consequential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。