提出多门控残差机制，稳定深层网络激活值且无需额外通信开销。
Multi-Gate Residuals
- 用评分与门控机制维持多路上下文信息
- 在大规模训练中提升模型性能，优于现有架构
- 适合需要高效部署的深度学习系统
尽管注意力残差在缓解深层残差层中激活值无界增长问题上已显成效，但不可避免带来显著通信开销。为克服此瓶颈，本文提出多门控残差（MGR），在不增加通信负担的前提下稳定激活尺度。该方法采用简洁的评分与门控机制保持多流上下文，并结合注意力池化从流状态中提取隐藏状态。实验证明，MGR适用于大规模训练与部署，在性能上明显优于现有架构。
原文摘要 · Abstract (English)
While Attention Residuals has shown some effectiveness in addressing the widespread issue of unbounded activation growth across deep residual layers, it inevitably incurs significant communication overhead. To circumvent this bottleneck, we propose Multi-Gate Residuals (MGR), which stabilizes activation scales without additional communication burden. It utilizes a straightforward scoring and gating mechanism to maintain multi-stream context, coupled with Attention Pooling to extract hidden states from the stream states. Empirical experiments demonstrate that MGR is practical for large-scale training and deployment, offering tangible performance improvements over existing architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。