动态连接提升注意力模型性能，用少量开销超越更大规模模型。
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
- 通过动态生成连接权重，增强跨层信息流动。
- 2.8B参数模型在预训练和下游任务中媲美6.9B甚至12B模型。
- 仅增加0.23%参数与0.4%计算，适合高效部署场景。
我们提出多路动态密集（MUDD）连接，一种简单而有效的方法，以解决残差连接的局限性并增强Transformer中的跨层信息流。与现有静态共享权重的密集连接方法不同，MUDD根据每个序列位置及解耦输入流（查询、键、值或残差）的隐藏状态动态生成连接权重。MUDD连接可无缝集成到任何Transformer架构中，形成MUDDFormer。大量实验表明，MUDDFormer在多种模型架构和规模的语言建模任务中显著优于标准Transformer，其性能相当于使用1.8倍至2.4倍计算量训练的Transformer。值得注意的是，MUDDPythia-2.8B在预训练困惑度和下游任务上达到Pythia-6.9B水平，甚至在五次提示设置下媲美Pythia-12B，而仅增加0.23%参数和0.4%计算量。代码与预训练模型已在https://github.com/Caiyun-AI/MUDDFormer 公开。
原文摘要 · Abstract (English)
We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective method to address the limitations of residual connections and enhance cross-layer information flow in Transformers. Unlike existing dense connection approaches with static and shared connection weights, MUDD generates connection weights dynamically depending on hidden states at each sequence position and for each decoupled input stream (the query, key, value or residual) of a Transformer block. MUDD connections can be seamlessly integrated into any Transformer architecture to create MUDDFormer. Extensive experiments show that MUDDFormer significantly outperforms Transformers across various model architectures and scales in language modeling, achieving the performance of Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B in five-shot settings, while adding only 0.23% parameters and 0.4% computation. Code in JAX and PyTorch and pre-trained models are available at https://github.com/Caiyun-AI/MUDDFormer .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。