只更新部分参数，大幅降低分布式训练的通信和内存开销。
Partial Parameter Updates for Efficient Distributed Training
- 每个节点仅更新固定参数子集，其余参数冻结。
- 32节点训练13亿参数模型，计算量与峰值内存显著下降。
- 适合资源受限场景下的大规模语言模型训练。
我们提出一种高效低通信的分布式训练方法。现有方法通过减少全局同步频率并增加本地更新次数来降低通信量。本文发现，通过限制反向传播——即每个节点仅更新固定参数子集而冻结其余参数——可显著提升效率。该策略大幅降低峰值内存占用和训练浮点运算量（FLOPs），同时全参数前向传播避免了跨节点激活值交换。在32个节点上训练13亿参数语言模型的实验表明，该方法在相同数据量与带宽预算下，达到与以往低通信方法相当的困惑度（perplexity），但训练计算量和峰值内存均更低。
原文摘要 · Abstract (English)
We introduce a memory- and compute-efficient method for low-communication distributed training. Existing methods reduce communication by performing multiple local updates between infrequent global synchronizations. We demonstrate that their efficiency can be significantly improved by restricting backpropagation: instead of updating all the parameters, each node updates only a fixed subset while keeping the remainder frozen during local steps. This constraint substantially reduces peak memory usage and training FLOPs, while a full forward pass over all parameters eliminates the need for cross-node activation exchange. Experiments on a $1.3$B-parameter language model trained across $32$ nodes show that our method matches the perplexity of prior low-communication approaches under identical token and bandwidth budgets while reducing training FLOPs and peak memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。