联邦学习中平均更新会削弱隐式正则化，导致泛化能力下降。
How Much Regularization Survives Averaging? Update Masking in Federated Learning
- 通过客户端独立掩码实现隐式正则化，利用参数更新噪声增强平坦极小值
- 联邦平均使正则化强度降低至原始水平的1/10，严重削弱泛化能力
- 数据异质性影响小，但关闭小批量采样可提升正则化效果至8.96倍
非独立同分布（non-IID）数据下的联邦学习旨在寻找平坦极小值以提升跨客户端泛化性能。现有方法借鉴集中式训练中的尖锐度感知最小化，但另一种无需显式设计的隐式正则化路径——即通过参数更新噪声自然引入正则项——尚未被引入联邦学习。本文揭示原因：掩码机制使优化器对进入尖锐方向产生惩罚，当各客户端使用独立掩码时，联邦平均会使该惩罚减弱至原始值的1/10（在CIFAR-10实验中为1.19/10），而若所有客户端共享相同掩码，则惩罚强度可恢复至原值的1.19~1.50倍（数据异质性变化千倍内），关闭小批量采样时甚至可达8.96倍。这些配置下正则化仍严重不足，无法有效使用。
原文摘要 · Abstract (English)
Federated learning on non-IID data seeks flat minima to generalize across clients, and existing methods borrow sharpness-aware minimization from centralized training. There is a second way to reach flat minima, in which the regularization comes for free from noise added to the parameter updates, and it has never been carried over to the federated setting as an implicit regularizer. We show the reason. Masking charges the optimizer for moving in sharp directions. We prove that when each client draws its own mask, federated averaging weakens that charge by exactly the cohort size, and that giving every client the same mask brings it back by a factor equal to the inverse gradient diversity of the cohort. In our experiment setting on CIFAR-10, that factor is 1.19 out of a possible 10. Turning off minibatch sampling raises it to 8.96, while changing data heterogeneity a thousandfold leaves it between 1.17 and 1.50. The configurations keeping the regularization train far too poorly to use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。