用低维子空间压缩激活值，实现模型并行的高效通信。
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
- 通过预定义低维子空间约束激活和梯度，实现双向压缩。
- 最高达99%压缩率，通信效率提升100倍，收敛无损失。
- 适合在低带宽设备上训练千亿参数模型，如80Mbps网络。
模型规模的扩大推动了深度学习的进步，但在去中心化训练中仍面临通信瓶颈。现有压缩技术在数据并行中有效，但难以应用于模型并行。与数据并行交换权重梯度不同，模型并行需压缩前向和反向传播中的激活值及梯度，压缩误差会累积。本文提出一种新型压缩算法，同时压缩前向与反向过程，可在几乎无额外内存或计算开销下实现高达99%的压缩率，且不损失收敛性。利用Transformer网络的递归结构，预先定义低维子空间以约束激活值和梯度，确保后续层可完全重建。该方法使通信效率提升最多100倍，支持在消费级互联网（最低80Mbps）连接的低端GPU上训练千亿参数模型，其收敛性能媲美拥有100Gbps连接的集中式数据中心系统。
原文摘要 · Abstract (English)
Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。