改进残差连接方式，让深层网络更高效地学习。
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
- 提出动态重分配残差连接的新方法，由数据自适应学习连接方式。
- 在多种模型上实现更快收敛，深度利用效率提升超过1%。
- 适合需要深度网络但想避免冗余计算的研究者使用。
深度扩展是现代基础模型成功的关键驱动力，但近期研究发现深层网络往往未被充分利用。本文从优化角度重新审视神经网络加深的默认机制——残差连接。严格分析证明，残差连接的布局会根本性影响收敛行为，甚至导致收敛速率的指数级差异。受此启发，本文提出自适应神经连接重分配（ANCRe），一种原则性强且轻量的框架，可参数化并从数据中学习残差连接关系。ANCRe以低于1%的计算与内存开销，动态重分配残差连接，有效提升网络深度利用率。在大规模语言模型、扩散模型和深层ResNet的预训练中，实验均显示收敛速度加快、性能提升，并显著增强深度效率。
原文摘要 · Abstract (English)
Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This paper revisits the default mechanism for deepening neural networks, namely residual connections, from an optimization perspective. Rigorous analysis proves that the layout of residual connections can fundamentally shape convergence behavior, and even induces an exponential gap in convergence rates. Prompted by this insight, we introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data. ANCRe adaptively reassigns residual connections with negligible computational and memory overhead ($<1\%$), while enabling more effective utilization of network depth. Extensive numerical tests across pre-training of large language models, diffusion models, and deep ResNets demonstrate consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。