arXiv:2510.06949cs.LGcs.AI2025-10被引 2

通过不均衡分配注意力头,提升Transformer信号聚焦能力。

Grouped Differential Attention

  • 将注意力头分为信号保留与噪声控制两组,动态分配数量
  • 适度不平衡分配使泛化性能显著优于对称基线
  • 仅复制信号头实现高效扩展,适合大规模模型设计

自注意力机制在现代Transformer中虽基础重要,但常将大量注意力投向冗余或噪声信息。差分注意力通过减法式注意力图分离信号与噪声,但需平衡头分配,限制了表达灵活性与可扩展性。为此,我们提出分组差分注意力(GDA),引入信号保留与噪声控制组间的非对称头分配:更多头用于信号提取,较少头用于噪声控制,并通过可控重复稳定后者(类似分组查询注意力,GQA)。该设计在极小计算开销下显著增强信号保真度。进一步提出组差异化增长策略,仅选择性复制信号关注头,实现高效容量扩展。大规模预训练与持续训练实验表明,适度不平衡的GDA在泛化与稳定性上均显著优于对称基线。结果表明,比例感知的头分配与选择性扩展为构建可扩展、高效计算的Transformer架构提供了有效且实用路径。

原文摘要 · Abstract (English)

The self-attention mechanism, while foundational to modern Transformer architectures, suffers from a critical inefficiency: it frequently allocates substantial attention to redundant or noisy context. Differential Attention addressed this by using subtractive attention maps for signal and noise, but its required balanced head allocation imposes rigid constraints on representational flexibility and scalability. To overcome this, we propose Grouped Differential Attention (GDA), a novel approach that introduces unbalanced head allocation between signal-preserving and noise-control groups. GDA significantly enhances signal focus by strategically assigning more heads to signal extraction and fewer to noise-control, stabilizing the latter through controlled repetition (akin to GQA). This design achieves stronger signal fidelity with minimal computational overhead. We further extend this principle to group-differentiated growth, a scalable strategy that selectively replicates only the signal-focused heads, thereby ensuring efficient capacity expansion. Through large-scale pretraining and continual training experiments, we demonstrate that moderate imbalance ratios in GDA yield substantial improvements in generalization and stability compared to symmetric baselines. Our results collectively establish that ratio-aware head allocation and selective expansion offer an effective and practical path toward designing scalable, computation-efficient Transformer architectures.

注意力机制Transformer模型压缩可扩展性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。