将鲁棒联邦学习转化为带误差梯度的优化问题,实现高效加速。
From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity
- 把鲁棒聚合建模为含加/乘性误差的不精确梯度优化
- 提出两种加速算法,通信复杂度显著低于现有方法
- 适用于存在恶意节点的分布式学习场景
标准联邦学习算法易受恶意节点(即拜占庭故障)影响。为解决此问题,研究者提出了鲁棒分布式学习算法,通常以鲁棒聚合替代参数平均。尽管已有通用条件可保证(随机)梯度下降的收敛性,但分析仍较零散,阻碍了更复杂鲁棒算法(如加速算法)的发展。本文表明,在标准通用假设下,拜占庭鲁棒分布式优化可转化为具有不精确梯度预言机(含加性和乘性误差项)的一般优化问题,这一领域正活跃研究。这使得可直接证明:在标准鲁棒聚合基础上使用梯度下降能获得拜占庭设置下的最优渐近误差。进一步,我们提出两种加速方案:第一种是基于Nesterov的加速方法,其证明源自对不精确梯度加速结果的应用;第二种基于相似性优化,服务器利用辅助损失函数逼近全局损失。两者均显著降低通信复杂度,理论与实证均验证了其优越性。
原文摘要 · Abstract (English)
Standard federated learning algorithms are vulnerable to adversarial nodes, a.k.a. Byzantine failures. To solve this issue, robust distributed learning algorithms have been developed, which typically replace parameter averaging by robust aggregations. While generic conditions on these aggregations exist to guarantee the convergence of (Stochastic) Gradient Descent (SGD), the analyses remain rather ad-hoc. This hinders the development of more complex robust algorithms, such as accelerated ones. In this work, we show that Byzantine-robust distributed optimization can, under standard generic assumptions, be cast as a general optimization with inexact gradient oracles (with both additive and multiplicative error terms), an active field of research. This allows for instance to directly show that GD on top of standard robust aggregation procedures obtains optimal asymptotic error in the Byzantine setting. Going further, we propose two optimization schemes to speed up the convergence. The first one is a Nesterov-type accelerated scheme whose proof directly derives from accelerated inexact gradient results applied to our formulation. The second one hinges on Optimization under Similarity, in which the server leverages an auxiliary loss function that approximates the global loss. Both approaches allow to drastically reduce the communication complexity compared to previous methods, as we show theoretically and empirically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。