提出新型联邦优化框架,用代理函数聚合替代参数聚合,提升分布式学习鲁棒性。
Federated Majorize-Minimization: Beyond Parameter Aggregation
- 以代理函数为优化单元,突破传统参数聚合局限
- 在数据异构和通信受限下仍保持收敛性
- 可扩展至最优传输等复杂任务,适用广泛
本文提出一种统一的随机优化算法设计方法,可稳健扩展至联邦学习场景。研究一类具有线性参数化代理函数的极大极小(MM)问题,涵盖(近端)梯度法、期望最大化算法及多种变分代理MM问题。我们证明该框架可导出统一算法——随机近似随机代理MM(\ exttt{SSMM}),包含已有随机MM方法作为特例。进一步将 exttt{SSMM}拓展至联邦设置,考虑数据异构、部分参与和通信约束等常见瓶颈,提出 exttt{QSMM}。其创新在于本地学习后聚合描述“代理上界函数”的信息,而非原始参数。最后,为展示该方法灵活性,我们将其用于设计联邦环境下最优传输映射的计算算法。
原文摘要 · Abstract (English)
This paper proposes a unified approach for designing stochastic optimization algorithms that robustly scale to the federated learning setting. Our work studies a class of Majorize-Minimization (MM) problems, which possesses a linearly parameterized family of majorizing surrogate functions. This framework encompasses (proximal) gradient-based algorithms for (regularized) smooth objectives, the Expectation Maximization algorithm, and many problems seen as variational surrogate MM. We show that our framework motivates a unifying algorithm called Stochastic Approximation Stochastic Surrogate MM (\SSMM), which includes previous stochastic MM procedures as special instances. We then extend \SSMM\ to the federated setting, while taking into consideration common bottlenecks such as data heterogeneity, partial participation, and communication constraints; this yields \QSMM. The originality of \QSMM\ is to learn locally and then aggregate information characterizing the \textit{surrogate majorizing function}, contrary to classical algorithms which learn and aggregate the \textit{original parameter}. Finally, to showcase the flexibility of this methodology beyond our theoretical setting, we use it to design an algorithm for computing optimal transport maps in the federated setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。