优化注意力机制梯度，提升训练效率与模型性能。
Scaling Bidirectional Spans and Span Violations in Attention Mechanism
- 通过非对称投影分解反向传播梯度为并行跨度与正交违例。
- 在WikiText-2上实现0.56%验证损失降低，证明方法有效性。
- 适合关注注意力机制优化与高效训练的开发者参考。
经典的 $O(N^2)$ Transformer 在序列建模中仍保持最优性能，其训练可通过缓解几何低效性进一步优化。本文提出一种优化框架，利用非对称投影将反向传播梯度分解为并行跨度与正交违例，同时保持前向传播 $QKV$ 结构不变。在多种分解与投影设置下进行一致实验验证，提供了强有力的理论证据：标准注意力梯度存在次优性。我们发现，有选择地放大这些组件，尤其是0阶双向并行跨度，能获得最有效的学习信号。在有限的WikiText-2数据集上，采用粗略配置即实现0.56%的验证损失下降，证实了该框架的基本有效性，并暗示在更大数据集和更深训练阶段具有显著提升潜力。
原文摘要 · Abstract (English)
The canonical $O(N^2)$ Transformer remains the empirical performance frontier in sequence modeling, and its training can be further optimized by addressing geometric inefficiency. We propose an optimization framework that leverages an asymmetric projection to decompose the backward-pass gradients into parallel spans and orthogonal violations, while keeping the canonical forward-pass $QKV$ structure intact. Through consistent experimental validation across various decomposition and projection setups, we provide strong theoretical evidence: the standard attention gradient is suboptimal. We demonstrated that selectively scaling these components, focusing primarily on $0^{th}$ order bidirectional parallel spans, yields the most effective learning signal. On the limited WikiText-2 dataset, and using a crude configuration, this method achieved a $0.56\%$ reduction in validation loss, confirming the framework's fundamental validity and suggesting significant potential gains on larger datasets and deeper training regimes
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。