动态量化加速大模型训练中的梯度同步，性能提升超34%。
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
- 针对多跳聚合设计动态量化方法，优化部分和表示
- 在多种模型与规模下实现最高34.2%的加速，精度接近基线
- 唯一实现高速训练同时保持近全精度准确率的方法
多跳全部归约是大规模模型训练的主流架构。随着训练规模扩大，网络常成为瓶颈，促使减少传输数据量。近期系统通过梯度量化显著加速训练过程。然而，这些方法未针对多跳聚合进行优化，其中梯度项在聚合路径上被多次部分求和。我们提出DynamiQ，一种连接量化最佳实践与多跳聚合的量化框架。DynamiQ引入新机制以更好表示部分和，并配套设计了压缩-累加-重压缩融合核以实现高效执行。我们扩展了PyTorch DDP以支持基于NCCL P2P的DynamiQ，在不同LLM、任务和规模下，其性能相比Omni-Reduce、THC及新兴标准MXFP4、MXFP6、MXFP8等最佳方法，平均提升达34.2%。此外，DynamiQ是唯一在所有测试中均保持近基线精度（如达到BF16基线的99.9%）且显著加速训练的方法。
原文摘要 · Abstract (English)
Multi-hop all-reduce is the de facto backbone of large model training. As the training scale increases, the network often becomes a bottleneck, motivating the reduction of the volume of transmitted data. Accordingly, recent systems have demonstrated significant acceleration of the training process using gradient quantization. However, these systems are not optimized for multi-hop aggregation, where entries are partially summed multiple times along their aggregation topology. We present DynamiQ, a quantization framework that bridges the gap between quantization best practices and multi-hop aggregation. DynamiQ introduces novel techniques to better represent partial sums, codesigned with a decompress accumulate recompress fused kernel to facilitate fast execution. We extend PyTorch DDP to support DynamiQ over NCCL P2P, and across different LLMs, tasks, and scales, we demonstrate consistent improvement of up to 34.2% over the best among state-of-the-art methods such as Omni-Reduce, THC, and emerging standards such as MXFP4, MXFP6, and MXFP8. Further, DynamiQ is the only evaluated method that consistently reaches near-baseline accuracy (e.g., 99.9\% of the BF16 baseline) and does so while significantly accelerating the training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。