用动态加权注意力增强Transformer残差连接，提速3倍且不增加参数
DeepCrossAttention: Supercharging Transformer Residual Connections
- 通过可学习的输入相关权重动态融合多层输出
- 相同精度下训练速度提升3倍，参数几乎不变
- 适合追求高效训练的NLP研究者与工程师
Transformer模型在多个领域取得成功，依赖于包括残差连接在内的多种架构创新。然而，传统残差连接简单地对前层输出求和，可能导致关键信息稀释。本文提出DeepCrossAttention（DCA），通过可学习的、依赖输入的权重动态组合层输出，使模型能选择性聚焦于任意前层中最相关的信息。此外，DCA引入深度交叉注意力，促进不同深度层间更丰富的交互。语言建模实验表明，DCA在给定训练时间内实现更低困惑度；同时，在保持相同模型质量的前提下，可实现最高3倍的加速，且参数增加可忽略。理论分析证实，当层秩总和与环境维度之比低于临界阈值时，DCA在准确率与模型大小之间提供了更优权衡。
原文摘要 · Abstract (English)
Transformer networks have achieved remarkable success across diverse domains, leveraging a variety of architectural innovations, including residual connections. However, traditional residual connections, which simply sum the outputs of previous layers, can dilute crucial information. This work introduces DeepCrossAttention (DCA), an approach that enhances residual learning in transformers. DCA employs learnable, input-dependent weights to dynamically combine layer outputs, enabling the model to selectively focus on the most relevant information in any of the previous layers. Furthermore, DCA incorporates depth-wise cross-attention, allowing for richer interactions between layers at different depths. Our language modeling experiments show that DCA achieves improved perplexity for a given training time. Moreover, DCA obtains the same model quality up to 3x faster while adding a negligible number of parameters. Theoretical analysis confirms that DCA provides an improved trade-off between accuracy and model size when the ratio of collective layer ranks to the ambient dimension falls below a critical threshold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。