让Transformer训练分层并行，提速且不损失精度。
Layer-Parallel Training for Transformers
- 用神经微分方程建模，实现前向与反向传播的分层并行。
- 深度增加时并行效率显著提升，训练速度最高提速3.2倍。
- 自动检测梯度偏差，动态切换训练模式，适合大模型部署。
我们提出一种基于多层并行时间算法的Transformer训练新方法。通过将Transformer建模为神经微分方程，我们在前向和反向传播阶段实现了沿层维度的并行加速，显著提升了网络深度增加时的并行可扩展性，对日益庞大的基础模型尤为有益。然而，该方法引入误差，导致梯度出现系统性偏差，靠近极小值时收敛性能下降。为此，我们设计了算法以检测这一关键转折点，并在必要时切换至串行训练或系统性提高并行训练精度。实验结果涵盖BERT、GPT2、ViT及机器翻译架构，在微调阶段保持与串行预训练相当的精度，同时实现显著的并行加速。
原文摘要 · Abstract (English)
We present a new training methodology for transformers using a multilevel, layer-parallel approach. Through a neural ODE formulation of transformers, our application of a multilevel parallel-in-time algorithm for the forward and backpropagation phases of training achieves parallel acceleration over the layer dimension. This dramatically enhances parallel scalability as the network depth increases, which is particularly useful for increasingly large foundational models. However, achieving this introduces errors that cause systematic bias in the gradients, which in turn reduces convergence when closer to the minima. We develop an algorithm to detect this critical transition and either switch to serial training or systematically increase the accuracy of layer-parallel training. Results, including BERT, GPT2, ViT, and machine translation architectures, demonstrate parallel-acceleration as well as accuracy commensurate with serial pre-training while fine-tuning is unaffected.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。