小模型用隐变量多头注意力，省内存还更快，质量几乎不变。
Latent Multi-Head Attention for Small Language Models
- 用隐变量压缩注意力机制，降低内存占用。
- 内存减少45%，验证损失仅升0.3%，性能接近完整模型。
- 适合部署在内存受限设备，如边缘端或移动端。
我们首次系统研究了小语言模型中的隐变量多头注意力（MLA），揭示了效率与质量间的有趣权衡。在10万条合成故事上训练3000万参数的GPT模型，对比三种架构：标准多头注意力（MHA）、MLA，以及带旋转位置编码的MLA（MLA+RoPE）。关键发现：当隐变量维度为一半（r = d/2）时，MLA+RoPE实现45%的KV缓存内存缩减，验证损失仅上升0.3%（基本等同于MHA质量），构成内存受限部署的帕累托优化。进一步表明，旋转位置编码对小模型中的MLA至关重要：无RoPE时，性能比标准注意力低3-5%；有RoPE时则反超2%。NVIDIA A100推理测试显示，r=d/2的MLA相比全秩MLA提速1.4倍，同时保持内存优势。GPT-4评估确认困惑度结果，我们的方法在语法、创意和一致性三项指标上得分最高（7.4/10）。代码与模型将在录用后发布。
原文摘要 · Abstract (English)
We present the first comprehensive study of latent multi-head attention (MLA) for small language models, revealing interesting efficiency-quality trade-offs. Training 30M-parameter GPT models on 100,000 synthetic stories, we benchmark three architectural variants: standard multi-head attention (MHA), MLA, and MLA with rotary positional embeddings (MLA+RoPE). Our key finding is that MLA+RoPE with half-rank latent dimensions (r = d/2) achieves a 45% KV-cache memory reduction while incurring only a 0.3% increase in validation loss (essentially matching MHA quality)- a Pareto improvement for memory constrained deployment. We further show that RoPE is crucial for MLA in small models: without it, MLA underperforms vanilla attention by 3-5%, but with RoPE, it surpasses vanilla by 2%. Inference benchmarks on NVIDIA A100 GPUs reveal that MLA with r=d/2 achieves a 1.4 times speedup over full-rank MLA while maintaining the memory savings. GPT-4 evaluations corroborate perplexity results, with ours achieving the highest quality scores (7.4/10) across grammar, creativity, and consistency metrics. Code and models will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。