提出Tucker注意力，用低秩分解压缩大模型注意力，参数更少性能不降。
Tucker Attention: A generalization of approximate attention mechanisms
- 基于张量分解思想，统一建模注意力权重，实现参数高效压缩。
- 在大模型和视觉模型上，参数量减少一个数量级，验证指标相当。
- 兼容Flash Attention与旋转位置编码，可覆盖原有多头注意力变体。
为降低多头自注意力（MHA）的内存开销，研究者提出了多种近似方法，如分组查询注意力（GQA）和多头潜在注意力（MLA），它们通过嵌入维度或注意力头上的特定低秩分解来实现。但从经典低秩逼近的角度看，这些方法具有非传统性，引发对它们实际逼近对象及表示低秩行为的疑问。本文提出一种关于自注意力层权重对象的广义视角与因子分解策略，构建出参数高效的方案——Tucker注意力。在大语言模型和视觉变换器（ViT）测试中，该方法相比GQA和MLA，在保持相近验证指标的前提下，参数量减少一个数量级。此外，Tucker注意力涵盖MHA、GQA、MLA作为特例，且完全兼容Flash Attention与旋转位置编码（RoPE）。该泛化策略揭示了MHA、GQA、MLA的实际秩水平，并进一步简化了MLA结构。
原文摘要 · Abstract (English)
The pursuit of reducing the memory footprint of the self-attention mechanism in multi-headed self attention (MHA) spawned a rich portfolio of methods, e.g., group-query attention (GQA) and multi-head latent attention (MLA). The methods leverage specialized low-rank factorizations across embedding dimensions or attention heads. From the point of view of classical low-rank approximation, these methods are unconventional and raise questions of which objects they really approximate and how to interpret the low-rank behavior of the resulting representations. To answer these questions, this work proposes a generalized view on the weight objects in the self-attention layer and a factorization strategy, which allows us to construct a parameter efficient scheme, called Tucker Attention. Tucker Attention requires an order of magnitude fewer parameters for comparable validation metrics, compared to GQA and MLA, as evaluated in LLM and ViT test cases. Additionally, Tucker Attention~encompasses GQA, MLA, MHA as special cases and is fully compatible with flash-attention and rotary position embeddings (RoPE). This generalization strategy yields insights of the actual ranks achieved by MHA, GQA, and MLA, and further enables simplifications for MLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。