一种可动态调节特征连接强度的新方法，提升大模型训练稳定性
Hyper-Connections
- 通过动态调整不同层间特征连接强度来优化梯度传播
- 在大语言模型预训练中显著优于残差连接，提升模型性能
- 适用于语言与视觉任务，尤其适合密集和稀疏模型
我们提出超连接（hyper-connections），一种简单而有效的替代残差连接的方法。该方法针对残差连接变体常见的梯度消失与表征坍塌之间的权衡问题进行了改进。理论上，超连接允许网络在不同深度动态调节特征间连接强度，并实现层的动态重组。我们在大语言模型的预训练任务中进行了实验，涵盖稠密与稀疏模型，结果表明超连接在性能上显著优于残差连接。此外，在视觉任务上的额外实验也显示出类似的性能提升。我们预期该方法将在广泛的AI问题中具有广泛应用价值。
原文摘要 · Abstract (English)
We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and sparse models, where hyper-connections show significant performance improvements over residual connections. Additional experiments conducted on vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across a wide range of AI problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。