提出首个中心点逼近下界,提升抗拜占庭联邦学习的鲁棒性。
Centroid Approximation for Byzantine-Tolerant Federated Learning
- 基于盒子有效性条件,给出中心点逼近的理论下界。
- 设计新算法实现√(2d)逼近,在凸有效性下证明下界紧致。
- 适用于去中心化网络,实验证明在梯度下降中有效。
联邦学习允许客户端在分布式训练中本地保存数据。近期研究明确了保证训练收敛所需的输入条件,通常采用平均作为聚合规则。本文关注联邦学习对拜占庭行为的鲁棒性,发现平均与分布式计算中的有效性条件间存在权衡。我们证明仅满足有效性条件不足以良好逼近平均;且即使逼近良好,实验中仍可能因拜占庭异常值而表现不佳。主要贡献是首次在盒子有效性假设下建立中心点逼近的下界:min{(n−t)/t, √d},其中 n 为客户端数,t 为拜占庭故障上限,d 为模型维度。同时提供上界 2 min{n, √d},并针对 n < d 情况进行新分析。此外,提出新算法在凸有效性下实现 √(2d) 逼近,证明现有文献下界紧致。所有边界均可在去中心化对等设置中达成。通过联邦随机梯度下降和联邦平均设置的实证评估验证理论结果。
原文摘要 · Abstract (English)
Federated learning allows each client to keep its data locally when training machine learning models in a distributed setting. Significant recent research established the requirements that the input must satisfy in order to guarantee convergence of the training loop. This line of work uses averaging as the aggregation rule for the training models. In particular, we are interested in whether federated learning is robust to Byzantine behavior, and observe and investigate a tradeoff between the average/centroid and the validity conditions from distributed computing. We show that the various validity conditions alone do not guarantee a good approximation of the average. Furthermore, we show that reaching good approximation does not give good results in experimental settings due to possible Byzantine outliers. Our main contribution is the first lower bound of $\min\{\frac{n-t}{t},\sqrt{d}\}$ on the centroid approximation under box validity that is often considered in the literature, where $n$ is the number of clients, $t$ the upper bound on the number of Byzantine faults, and $d$ is the dimension of the machine learning model. We complement this lower bound by an upper bound of $2\min\{n,\sqrt{d}\}$, by providing a new analysis for the case $n<d$. In addition, we present a new algorithm that achieves a $\sqrt{2d}$-approximation under convex validity, which also proves that the existing lower bound in the literature is tight. We show that all presented bounds can also be achieved in the distributed peer-to-peer setting. We complement our analytical results with empirical evaluations in federated stochastic gradient descent and federated averaging settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。