用递归更新替代堆叠层,提升训练速度与效率
SCORE: Replacing Layer Stacking with Contractive Recurrent Depth
- 用微分方程启发的递归机制替代多层堆叠
- 训练收敛更快,参数量减少,计算开销低
- 适合追求轻量化与高效训练的深度模型
残差连接是现代深度神经网络的核心,支持稳定优化和深层信息流动。本文提出SCORE(Skip-Connection ODE Recurrent Embedding),一种替代传统层堆叠的离散递归方法。SCORE通过一个共享神经模块,采用基于常微分方程(ODE)的收缩更新机制:ht+1 = (1 - dt) * ht + dt * F(ht),实现深度由迭代次数控制的渐进优化。该过程可视为逐次精炼,步长dt显式调控稳定性与更新幅度。与连续神经ODE不同,SCORE采用固定迭代次数和标准反向传播,无需求解器或伴随法。在图神经网络(ESOL分子溶解度)、多层感知机及基于Transformer的语言模型(nanoGPT)上评估显示,SCORE普遍加快收敛速度,并提升训练效率。通过权重共享,模型参数显著减少。实践中,简单欧拉积分在性能与开销间达到最佳平衡,高阶积分仅带来边际收益但增加计算成本。结果表明,受控的递归深度结合收缩残差更新,为深度网络提供了一种轻量且有效的层堆叠替代方案。
原文摘要 · Abstract (English)
Residual connections are central to modern deep neural networks, enabling stable optimization and efficient information flow across depth. In this work, we propose SCORE (Skip-Connection ODE Recurrent Embedding), a discrete recurrent alternative to classical layer stacking. Instead of composing multiple independent layers, SCORE iteratively applies a single shared neural block using an ODE (Ordinary Differential Equation)-inspired contractive update: ht+1 = (1 - dt) * ht + dt * F(ht) This formulation can be interpreted as a depth-by-iteration refinement process, where the step size dt explicitly controls stability and update magnitude. Unlike continuous Neural ODE approaches, SCORE uses a fixed number of discrete iterations and standard backpropagation without requiring ODE solvers or adjoint methods. We evaluate SCORE across graph neural networks (ESOL molecular solubility), multilayer perceptrons, and Transformer-based language models (nanoGPT). Across architectures, SCORE generally improves convergence speed and often accelerates training. SCORE is reducing parameter count through shared weights. In practice, simple Euler integration provides the best trade-off between computational cost and performance, while higher-order integrators yield marginal gains at increased compute. These results suggest that controlled recurrent depth with contractive residual updates offers a lightweight and effective alternative to classical stacking in deep neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。