提出抗拜占庭攻击且通信高效的分布式训练方法,解决数据异构下误差不降问题。
Byzantine-Robust and Communication-Efficient Distributed Training: Compressive and Cyclic Gradient Coding
- 用循环梯度编码实现计算冗余,提升系统鲁棒性
- 理论证明收敛性,解决异构数据下误差不降难题
- 支持压缩传输,适合资源受限场景使用
本文研究在拜占庭攻击与通信约束下的分布式训练问题。现有方法虽具备一定抗攻击能力,但在设备间数据异构导致局部梯度差异较大时,解误差无法下降。为此,我们提出基于循环梯度编码的分布式训练方法(LAD):训练前将全量数据分配给各设备,在每轮迭代中通过循环编码冗余分配计算任务。诚实设备在固定数据子集上计算局部梯度并编码后上传;服务器使用鲁棒聚合规则融合来自诚实设备的编码向量及可能的恶意消息。利用设备间计算冗余,理论分析表明LAD具有更强的抗拜占庭攻击能力,显著降低解误差。进一步提出通信高效变体Com-LAD,结合压缩机制,在受限环境下有效降低通信开销。数值实验验证了所提方法在提升鲁棒性与通信效率方面的有效性。
原文摘要 · Abstract (English)
In this paper, we study the problem of distributed training (DT) under Byzantine attacks with communication constraints. While prior work has developed various robust aggregation rules at the server to enhance robustness to Byzantine attacks, the existing methods suffer from a critical limitation in that the solution error does not diminish when the local gradients sent by different devices vary considerably, as a result of data heterogeneity among the subsets held by different devices. To overcome this limitation, we propose a novel DT method, cyclic gradient coding-based DT (LAD). In LAD, the server allocates the entire training dataset to the devices before training begins. In each iteration, it assigns computational tasks redundantly to the devices using cyclic gradient coding. Each honest device then computes local gradients on a fixed number of data subsets and encodes the local gradients before transmitting to the server. The server aggregates the coded vectors from the honest devices and the potentially incorrect messages from Byzantine devices using a robust aggregation rule. Leveraging the redundancy of computation across devices, the convergence performance of LAD is analytically characterized, demonstrating improved robustness against Byzantine attacks and significantly lower solution error. Furthermore, we extend LAD to a communication-efficient variant, compressive and cyclic gradient coding-based DT (Com-LAD), which further reduces communication overhead under constrained settings. Numerical results validate the effectiveness of the proposed methods in enhancing both Byzantine resilience and communication efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。