提出正交自注意力机制，解决无残差连接的Transformer训练不稳问题。
Orthogonal Self-Attention
- 通过矩阵指数将查询键映射为正交注意力矩阵，避免梯度崩溃。
- 计算复杂度与内存开销随序列长度线性增长，可高效实现。
- 设计了保证雅可比矩阵良好条件数的初始化方法，利于模型训练。
Softmax自注意力（SSA）是Transformer架构的核心组件。然而，在无需残差连接的架构中，近期研究指出其存在固有不稳定性，会导致秩坍缩和病态雅可比矩阵。本文提出一种新型注意力机制——正交自注意力（OSA），旨在规避上述问题，使无需残差连接与归一化层的非因果Transformer更易训练。具体而言，OSA通过将查询-键值构成的斜对称矩阵经矩阵指数映射，参数化出正交注意力矩阵。我们证明该方法可实际实现，利用查询-键值的低秩结构，使OSA的计算复杂度和内存开销随序列长度呈线性增长。此外，我们推导出一种初始化方案，理论上保证了OSA的雅可比矩阵具有良好条件数。
原文摘要 · Abstract (English)
Softmax Self-Attention (SSA) is a key component of Transformer architectures. However, when utilised within skipless architectures, which aim to improve representation learning, recent work has highlighted the inherent instability of SSA due to inducing rank collapse and poorly-conditioned Jacobians. In this work, we design a novel attention mechanism: Orthogonal Self-Attention (OSA), which aims to bypass these issues with SSA, in order to allow for (non-causal) Transformers without skip connections and normalisation layers to be more easily trained. In particular, OSA parametrises the attention matrix to be orthogonal via mapping a skew-symmetric matrix, formed from query-key values, through the matrix exponential. We show that this can be practically implemented, by exploiting the low-rank structure of our query-key values, resulting in the computational complexity and memory cost of OSA scaling linearly with sequence length. Furthermore, we derive an initialisation scheme for which we prove ensures that the Jacobian of OSA is well-conditioned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。