减少Transformer的投影数量，能显著降低内存占用且不影响性能。
Do Transformers Need Three Projections? Systematic Study of QKV Variants
- 用共享键值投影替代三个独立投影，简化注意力计算。
- 键值共享使缓存减少50%，语言建模困惑度仅下降3.1%。
- 结合分组查询，缓存压缩可达96.9%，适合设备端部署。
Transformer已成为各类AI任务的标准架构，其查询、键、值（QKV）注意力机制起核心作用。然而，这三个投影各自的贡献及省略部分的影响仍不明确。本文系统评估三种投影共享策略：a) Q-K=V（共享键值），b) Q=K-V（共享查询键），c) Q=K=V（单一投影）。后两种产生对称注意力图，为此引入二维位置编码实现非对称注意力。在合成任务、视觉（MNIST、CIFAR、TinyImageNet、异常检测）和语言建模（3亿与12亿参数模型，基于100亿词训练）上的实验表明，所提模型表现与标准QKV Transformer相当甚至更优。在语言建模中，Q-K=V共享实现50%缓存缩减，困惑度仅增加3.1%。关键发现：投影共享与头共享（GQA/MQA）互补——结合GQA-4可实现87.5%缓存压缩，搭配MQA达96.9%，支持实际设备端推理。结果表明，Q-K=V保持性能是因为键与值可共享表征空间，且注意力处于低秩状态；而Q=K-V破坏注意力方向性。本研究系统揭示了注意力中权重共享这一被忽视的优化路径，具有可量化的推理内存收益，尤其适用于边缘部署。代码已开源。
原文摘要 · Abstract (English)
Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual contribution of these three projections and the impact of omitting some remain poorly understood. We systematically evaluate three projection sharing constraints: a) Q-K=V (shared key-value), b) Q=K-V (shared query-key), and c) Q=K=V (single projection). The last two variants produce symmetric attention maps; to address this, we also explore asymmetric attention via 2D positional encodings. Through experiments spanning synthetic tasks, vision (MNIST, CIFAR, TinyImageNet, anomaly), and language modeling (300M and 1.2B parameter models on 10B tokens), we discovered that our transformers perform on par or occasionally better than the QKV transformer. In language modeling, Q-K=V projection sharing achieves 50% KV cache reduction with only 3.1% perplexity degradation. Crucially, projection sharing is complementary to head sharing (GQA/MQA): combining Q-K=V with GQA-4 yields 87.5% cache reduction, while Q-K=V + MQA achieves 96.9%, enabling practical on-device inference. We show that Q-K=V preserves quality because keys and values can occupy similar representational spaces and attention operates in a low-rank regime, whereas Q=K-V breaks attention directionality. Our results systematically characterize projection sharing as an underexplored instance of weight tying in attention, with direct, quantifiable inference memory benefits, particularly valuable for edge deployment. The code is publicly available at https://github.com/Brainchip-Inc/Do-Transformers-Need-3-Projections
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。