提出并行化反向传播算法，加速深度模型训练。
Accelerated Training through Iterative Gradient Propagation Along the Residual Path
- 通过残差路径迭代累积梯度，实现并行反向传播。
- 在多种模型上实验，训练速度提升显著且性能损失小。
- 适用于ResNet、Transformer等主流架构，适合大规模训练场景。
尽管反向传播是深度学习的基石，但其固有的顺序性限制了超深模型的可扩展性。此类模型曾因梯度消失导致收敛困难，后通过残差连接解决。如今这类结构已广泛应用于现代网络架构。然而，反向传播的计算开销仍是主要负担，占训练时间绝大部分。我们利用类似残差的结构设计，提出高速公路反向传播（Highway-BP），一种可并行的迭代算法，通过交替地沿残差路径累积梯度估计，并并行反向传播至各层，近似原始反向传播。该算法源于梯度按所有路径分解的思想，可适配多种常见架构，包括ResNets、Transformer和循环神经网络。在大量任务与模型上的实证研究表明，Highway-BP可在性能几乎不变的前提下实现显著加速。
原文摘要 · Abstract (English)
Despite being the cornerstone of deep learning, backpropagation is criticized for its inherent sequentiality, which can limit the scalability of very deep models. Such models faced convergence issues due to vanishing gradient, later resolved using residual connections. Variants of these are now widely used in modern architecture. However, the computational cost of backpropagation remains a major burden, accounting for most of the training time. Taking advantage of residual-like architectural designs, we introduce Highway backpropagation, a parallelizable iterative algorithm that approximates backpropagation, by alternatively i) accumulating the gradient estimates along the residual path, and ii) backpropagating them through every layer in parallel. This algorithm is naturally derived from a decomposition of the gradient as the sum of gradients flowing through all paths and is adaptable to a diverse set of common architectures, ranging from ResNets and Transformers to recurrent neural networks. Through an extensive empirical study on a large selection of tasks and models, we evaluate Highway-BP and show that major speedups can be achieved with minimal performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。