通过解耦梯度计算，大幅提升深度网络训练速度。
SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism
- 将长梯度流拆分为多个短流，实现分层并行计算
- 相比标准BP，训练吞吐量显著提升，超越GPipe和AL等方法
- 适合追求高效训练的团队，尤其适用于大规模模型部署
在企业信息系统中应用大规模AI模型常受高昂训练成本和漫长开发周期制约，管理挑战突出。标准端到端反向传播（BP）算法虽是现代AI的核心，但也是深度网络训练效率低下的根源。本文提出一种新训练方法——监督对比并行学习（SCPL），通过解耦反向传播，将长梯度流转化为多个短梯度流，使不同层的参数梯度可同时计算，实现更优的模型并行性，显著提升训练吞吐量。实验对比表明，SCPL在效率与效果上均优于标准BP、Early Exit、GPipe及当前最优的解耦反向传播方法AL。该方法有效缓解了训练中的根本性能瓶颈，为组织以更低成本、更高敏捷性开发与部署先进信息系统提供了可行路径。代码已开源，便于复现。https://github.com/minyaho/scpl/
原文摘要 · Abstract (English)
Adopting large-scale AI models in enterprise information systems is often hindered by high training costs and long development cycles, posing a significant managerial challenge. The standard end-to-end backpropagation (BP) algorithm is a primary driver of modern AI, but it is also the source of inefficiency in training deep networks. This paper introduces a new training methodology, Supervised Contrastive Parallel Learning (SCPL), that addresses this issue by decoupling BP and transforming a long gradient flow into multiple short ones. This design enables the simultaneous computation of parameter gradients in different layers, achieving superior model parallelism and enhancing training throughput. Detailed experiments are presented to demonstrate the efficiency and effectiveness of our model compared to BP, Early Exit, GPipe, and Associated Learning (AL), a state-of-the-art method for decoupling backpropagation. By mitigating a fundamental performance bottleneck, SCPL provides a practical pathway for organizations to develop and deploy advanced information systems more cost-effectively and with greater agility. The experimental code is released for reproducibility. https://github.com/minyaho/scpl/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。