将普通大模型转为低显存注意力机制，仅用0.6%数据就能保持性能
Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
- 通过移除部分旋转位置编码和低秩近似，实现从标准注意力到潜变量注意力的高效迁移
- 在仅0.3%-0.6%训练数据下恢复原模型性能，7B模型KV缓存压缩92.19%
- 适合追求推理效率与显存优化的模型部署者，尤其适配量化等压缩技术
多头潜变量注意力（MLA）是DeepSeek提出的一种新架构，通过将键值缓存压缩为潜向量，显著降低推理成本。相比标准的多头注意力（MHA）及其变体（如分组查询注意力GQA），MLA具有明显优势。本文提出首个数据高效的微调方法（MHA2MLA），实现无需从头训练即可将预训练模型（如Llama）快速适配至MLA。该方法包含两项关键设计：针对部分旋转位置编码（partial-RoPE），移除对注意力分数贡献较小的查询与键的旋转编码；针对低秩近似，基于预训练的键值参数引入联合奇异值分解（SVD）近似。这些策略使模型仅需0.3%至0.6%的微调数据即可恢复性能。例如，Llama2-7B模型的KV缓存大小减少92.19%，长文本评测集LongBench性能仅下降0.5%，且可无缝集成量化等压缩技术。
原文摘要 · Abstract (English)
Multi-head Latent Attention (MLA) is an innovative architecture proposed by DeepSeek, designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector. Compared to MLA, standard LLMs employing Multi-Head Attention (MHA) and its variants such as Grouped-Query Attention (GQA) exhibit significant cost disadvantages. Enabling well-trained LLMs (e.g., Llama) to rapidly adapt to MLA without pre-training from scratch is both meaningful and challenging. This paper proposes the first data-efficient fine-tuning method for transitioning from MHA to MLA (MHA2MLA), which includes two key components: for partial-RoPE, we remove RoPE from dimensions of queries and keys that contribute less to the attention scores, for low-rank approximation, we introduce joint SVD approximations based on the pre-trained parameters of keys and values. These carefully designed strategies enable MHA2MLA to recover performance using only a small fraction (0.3% to 0.6%) of the data, significantly reducing inference costs while seamlessly integrating with compression techniques such as KV cache quantization. For example, the KV cache size of Llama2-7B is reduced by 92.19%, with only a 0.5% drop in LongBench performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。