用分块方式改进残差连接，降低内存开销同时提升训练效果。
Frac-Connections: Fractional Extension of Hyper-Connections
- 将隐藏状态分块处理，避免扩展宽度增加内存
- 70亿参数模型在3万亿词上训练，性能显著优于传统残差连接
- 适合大规模语言模型训练，尤其关注内存效率的场景
残差连接是现代深度学习架构的核心，有助于缓解梯度消失问题。超连接（Hyper-Connections）通过在不同深度引入多条连接路径，解决了梯度消失与表征坍缩之间的权衡问题，但其代价是增加了内存访问开销。本文提出一种新方法Frac-Connections，不扩展隐藏状态宽度，而是将其划分为多个部分，保留了超连接的部分优势的同时降低了内存消耗。我们在语言任务上进行了大规模实验，最大规模为在3万亿标记上训练的70亿参数MoE模型，结果表明Frac-Connections显著优于标准残差连接。
原文摘要 · Abstract (English)
Residual connections are central to modern deep learning architectures, enabling the training of very deep networks by mitigating gradient vanishing. Hyper-Connections recently generalized residual connections by introducing multiple connection strengths at different depths, thereby addressing the seesaw effect between gradient vanishing and representation collapse. However, Hyper-Connections increase memory access costs by expanding the width of hidden states. In this paper, we propose Frac-Connections, a novel approach that divides hidden states into multiple parts rather than expanding their width. Frac-Connections retain partial benefits of Hyper-Connections while reducing memory consumption. To validate their effectiveness, we conduct large-scale experiments on language tasks, with the largest being a 7B MoE model trained on up to 3T tokens, demonstrating that Frac-Connections significantly outperform residual connections.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。