通过分组查询专家机制，让注意力更省算力且保持精度。
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
- 每组查询头由路由器动态选k个专家，只激活部分计算
- 在30B token预算下，仅用一半查询头达到与全激活相同效果
- 适合需要长序列推理的高效模型部署场景
自注意力是Transformer性能的核心，但在长序列下因成对标记交互呈二次增长而成为最耗时部分。标准密集注意力对每个标记使用相同注意力头，无论其难度或信息量，这种统一激活会浪费算力，尤其在序列变长时。我们提出分组查询专家（GQE），在分组查询注意力（GQA）基础上引入专家混合层。每个GQA组内，路由器为每标记选择k个查询头专家，而所有键值（KV）头保持密集不变。因此，GQE保留了GQA的KV缓存优势，仅减少活跃查询头的计算量。在固定30B token预算、参数量2.5亿的情况下，GQE在下游任务中精度与全激活的GQA基线相当，但每标记激活的查询头数量减半。
原文摘要 · Abstract (English)
Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。