发现权重更新99%在精度下降后无意义,提出仅传有效更新的通信优化方法。
Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL
- 只传输能影响下一轮前向计算的权重更新,避免无效通信
- 在宽带受限网络上实现超100倍的通信压缩,权重重建完全一致
- 适合大规模语言模型分布式训练中的高效通信场景
在带宽受限的大语言模型分布式强化学习后训练中,权重同步和梯度同步是两大瓶颈。我们发现,在标准训练与推理使用的BF16精度下,约99%的每步权重更新在舍入后变得不可见。原因在于典型强化学习学习率下,Adam更新常低于本地BF16舍入阈值。基于此,我们提出计算可见稀疏化原则:仅传输会改变下一前向传播的更新。PULSE(Precision-gated Updates for Low-precision Sparse Exchange)将该原则转化为两个算法:PULSESync从训练器向推理工作节点发送无损稀疏BF16权重块;PULSELoCo对DiLoCo风格的FP32伪梯度同步进行稀疏化并结合误差反馈。在商用宽带受限网络上,PULSESync使权重同步通信量降低超过100倍,且可位级精确重建训练器权重。PULSELoCo在四个模型上表现媲美DiLoCo,同时相比DiLoCo减少17倍以上训练器间通信,相比DDP在最大设置下减少超过100倍。
原文摘要 · Abstract (English)
Bandwidth-constrained distributed reinforcement learning (RL) post-training of large language models is bottlenecked by two channels: weight synchronization from trainers to inference workers, and gradient or pseudo-gradient synchronization across trainers. We find that approximately 99% of per-step weight updates are invisible after the BF16 cast used by standard training and inference forward passes. We explain this sparsity by showing that, at typical RL post-training learning rates, Adam updates often fall below the local BF16 rounding threshold. We turn this observation into an algorithmic principle called compute-visible sparsification: transmit only updates that would change the next forward pass. PULSE (Precision-gated Updates for Low-precision Sparse Exchange) turns this principle into two communication algorithms: PULSESync sends lossless sparse BF16 weight patches from trainers to inference workers, and PULSELoCo sparsifies DiLoCo-style FP32 pseudo-gradient synchronization with error feedback. Over bandwidth-constrained commodity networks, PULSESync cuts weight-synchronization communication by over 100x while reconstructing trainer weights bit-identically. PULSELoCo matches DiLoCo across four models while reducing trainer-to-trainer communication by over 17x versus DiLoCo and over 100x versus DDP in the largest evaluated setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。