用历史梯度预测当前梯度，仅传差异部分，大幅减少分布式训练通信量。
Temporal Predictive Coding for Gradient Compression in Distributed Learning
- 基于历史梯度线性预测当前梯度，系数通过最小二乘优化。
- 仅当预测残差超过阈值时才传输，显著降低通信开销。
- 适合高通信成本场景，尤其适用于资源受限的分布式学习系统。
本文提出一种基于预测的梯度压缩方法，用于分布式学习中的事件触发通信。目标是利用本地梯度的时间相关性，减少分布式代理向参数服务器传输的信息量。采用线性预测器，将历史梯度组合以预测当前梯度，其系数通过求解最小二乘问题优化。每轮迭代中，各代理向服务器发送预测器系数，使服务器可计算预测梯度；仅当真实梯度与预测梯度之差（即预测残差）的范数超过阈值时，才传输该残差。若省略此步骤，服务器直接使用预测梯度作为估计。该设计在文献对比中表现出显著性能提升，实现了更低通信成本下的收敛。
原文摘要 · Abstract (English)
This paper proposes a prediction-based gradient compression method for distributed learning with event-triggered communication. Our goal is to reduce the amount of information transmitted from the distributed agents to the parameter server by exploiting temporal correlation in the local gradients. We use a linear predictor that \textit{combines past gradients to form a prediction of the current gradient}, with coefficients that are optimized by solving a least-square problem. In each iteration, every agent transmits the predictor coefficients to the server such that the predicted local gradient can be computed. The difference between the true local gradient and the predicted one, termed the \textit{prediction residual, is only transmitted when its norm is above some threshold.} When this additional communication step is omitted, the server uses the prediction as the estimated gradient. This proposed design shows notable performance gains compared to existing methods in the literature, achieving convergence with reduced communication costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。