通过剪枝与稀疏压缩结合，显著提升分布式训练通信效率。
PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning
- 先剪枝使权重和梯度变稀疏，再进行轻量级压缩通信。
- 在带宽受限下，训练吞吐提升1.25至8.72倍。
- 适合大规模视觉与语言模型的高效分布式训练。
大规模深度神经网络在各类任务中表现优异。随着模型和数据集增长，分布式训练耗时剧增，需更大集群。主要瓶颈在于梯度聚合开销。尽管梯度压缩与稀疏集体通信常被用于缓解网络负载，但许多压缩方案未能同时实现加速且保持精度。本文提出PacTrain框架，通过结合剪枝与稀疏梯度压缩，加速分布式训练。主动剪枝使模型权重和梯度稀疏，并通过全局共享梯度稀疏性信息，在不损害精度的前提下实现轻量级压缩通信。实验表明，该压缩策略接近最优，兼容all-reduce原语。在带宽受限条件下，对代表性视觉与语言模型训练任务，相比现有压缩系统，训练吞吐提升1.25至8.72倍。
原文摘要 · Abstract (English)
Large-scale deep neural networks (DNN) exhibit excellent performance for various tasks. As DNNs and datasets grow, distributed training becomes extremely time-consuming and demands larger clusters. A main bottleneck is the resulting gradient aggregation overhead. While gradient compression and sparse collective communication techniques are commonly employed to alleviate network load, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. This paper introduces PacTrain, a novel framework that accelerates distributed training by combining pruning with sparse gradient compression. Active pruning of the neural network makes the model weights and gradients sparse. By ensuring the global knowledge of the gradient sparsity among all distributed training workers, we can perform lightweight compression communication without harming accuracy. We show that the PacTrain compression scheme achieves a near-optimal compression strategy while remaining compatible with the all-reduce primitive. Experimental evaluations show that PacTrain improves training throughput by 1.25 to 8.72 times compared to state-of-the-art compression-enabled systems for representative vision and language models training tasks under bandwidth-constrained conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。