针对联邦学习中的异常噪声，提出新型降噪算法提升模型收敛速度。
Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm
- 结合动量与非线性映射，抑制梯度噪声的极端波动。
- 在非凸目标下收敛率达O(K^(-(p-1)/(2p-1)))，强凸时接近O(K^(-1+1/p))。
- 适合无线网络和物联网场景下的大规模联邦学习应用。
联邦学习（FL）是一种新兴的分布式机器学习范式，使本地设备在数据不集中、保持隐私的前提下协同训练全局模型。本文提出一种基于方差减少的算法VRA-FedSGD，用于应对大规模机器学习中普遍存在的重尾梯度噪声与通信噪声，此类噪声常见于无线网络和物联网部署。VRA-FedSGD采用动量方差减少技术结合非线性映射以缓解重尾梯度噪声，并通过方差减少聚合机制抑制重尾通信噪声。在均值意义下，对于非凸目标函数，其收敛速率为$/mathcal{O}ig(K^{-(p-1)/(2p-1)}ig)$，其中$ p $为重尾噪声的尾指数；在几乎必然意义下，对于强凸目标函数，收敛速率可达$ ilde{ ext{O}}ig(K^{-(1-1/(p-ε)})ig)$，其中$ ε $为任意小常数。基于真实世界数据的逻辑回归问题模拟实验验证了VRA-FedSGD的有效性。
原文摘要 · Abstract (English)
Federated learning (FL) is an emerging distributed machine learning paradigm that enables local devices to jointly train a global model while keeping data decentralized and private. We propose a variance-reduction based algorithm, VRA-FedSGD, for FL in the presence of heavy-tailed gradient noise and communication noise, where these noises are prevalent in large-scale machine learning over wireless networks and Internet of Things deployments. VRA-FedSGD employs a momentum variance reduction technique together with a nonlinear mapping to mitigate heavy-tailed gradient noise, and uses a variance-reduced aggregation mechanism to suppress heavy-tailed communication noise. In the mean sense, VRA-FedSGD achieves a convergence rate of {\small$\mathcal{O}\left(K^{-(p-1)/(2p-1)}\right)$} for nonconvex objective functions, where $p$ is the tail index of heavy-tailed noise. In the almost sure sense, VRA-FedSGD achieves a convergence rate of $\tilde{\mathcal{O}}\left(K^{-(1-1/(p-ε))}\right)$ for strongly convex objective functions, where $ε$ is an arbitrarily small constant. Simulated experiments on a logistic regression problem with real-world data verify the effectiveness of VRA-FedSGD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。