通过部分同步激活，让大模型训练通信量减半且精度不降。
Tensor-Parallelism with Partially Synchronized Activations
- 设计新型架构CAAT-Net，仅部分同步激活减少通信开销。
- 70亿参数模型通信量降低50%，预训练精度几乎不变。
- 适用于多种模型规模，加速训练与推理，部分场景还能提升验证损失。
大规模语言模型（LLMs）使用张量并行训练和推理时，需大量通信以同步激活值。我们发现，只需对现有流程做少量调整，即可在不完全同步激活的情况下训练LLM，从而显著降低带宽需求。为此提出「通信感知的张量并行架构」（CAAT-Net）。我们训练了一个70亿参数的CAAT-Net模型，结果显示，在几乎所有评估基准上，张量并行通信量最多可减少50%，且预训练精度无明显下降。此外，我们还测试了1.1亿和13000万参数的小模型，验证了方法的鲁棒性与可扩展性。在某些场景下，减少通信甚至能改善验证损失。最后，我们在不同设置和模型规模下证明了CAAT-Net能有效加速训练与推理工作负载。
原文摘要 · Abstract (English)
Training and inference of Large Language Models (LLMs) with tensor-parallelism requires substantial communication to synchronize activations. Our findings suggest that with a few minor adjustments to current practices, LLMs can be trained without fully synchronizing activations, reducing bandwidth demands. We name this "Communication-Aware Architecture for Tensor-parallelism" (CAAT-Net). We train a 7B parameter CAAT-Net model and show that tensor-parallel communication can be reduced by up to 50% with no significant drop in pretraining accuracy across nearly all evaluated benchmarks. We also experiment with smaller 130M and 1.1B models to show the robustness and scalability of our method. We find that, in some scenarios, validation loss can even improve when reducing communication. Finally, we demonstrate how CAAT-Net accelerates both training and inference workloads across various settings and model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。