针对联邦学习中数据异构导致的泛化难题，提出三种低通信开销优化算法。
Communication-Efficient Federated Group Distributionally Robust Optimization
- 采用分组分布鲁棒优化框架，结合顶K损失与KL正则化降低通信开销
- 通信复杂度降至O(1/ε⁴)或O(1/ε³)，优于传统方法
- 适用于自然语言处理与计算机视觉等真实场景，支持Adam型本地更新
联邦学习因客户端间数据量和分布差异大，易影响模型在多种分布下的泛化能力。现有基于分组分布鲁棒优化（GDRO）的方法常导致高通信与样本复杂度。本文提出面向通信高效的联邦分组分布鲁棒优化（FGDRO）算法：首先设计FGDRO-CVaR，优化平均前K个损失，将通信复杂度降为O(1/ε⁴)，其中ε为期望精度；其次提出FGDRO-KL，通过KL正则化进一步将通信复杂度降至O(1/ε³)；最后引入FGDRO-KL-Adam，在保持相同通信成本的同时，利用Adam型本地更新，实测表现优于标准SGD型更新。算法在多个真实任务（如NLP、计算机视觉）上验证有效。
原文摘要 · Abstract (English)
Federated learning faces challenges due to the heterogeneity in data volumes and distributions at different clients, which can compromise model generalization ability to various distributions. Existing approaches to address this issue based on group distributionally robust optimization (GDRO) often lead to high communication and sample complexity. To this end, this work introduces algorithms tailored for communication-efficient Federated Group Distributionally Robust Optimization (FGDRO). Our contributions are threefold: Firstly, we introduce the FGDRO-CVaR algorithm, which optimizes the average top-K losses while reducing communication complexity to $O(1/ε^4)$, where $ε$ denotes the desired precision level. Secondly, our FGDRO-KL algorithm is crafted to optimize KL regularized FGDRO, cutting communication complexity to $O(1/ε^3)$. Lastly, we propose FGDRO-KL-Adam to utilize Adam-type local updates in FGDRO-KL, which not only maintains a communication cost of $O(1/ε^3)$ but also shows potential to surpass SGD-type local steps in practical applications. The effectiveness of our algorithms has been demonstrated on a variety of real-world tasks, including natural language processing and computer vision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。