用合成数据提升单轮联邦学习性能,解决数据异构问题
One-shot Federated Learning via Synthetic Distiller-Distillate Communication
- 用生成器直接从本地数据合成知识传递,减少信息损失
- 在复杂真实数据集上性能比最优基线高2.6倍
- 适合数据分布差异大、需单轮通信的场景
单轮联邦学习(One-shot FL)通过单次通信实现模型协同训练,虽具备高效与隐私保护优势,但常牺牲模型性能。现有方法多依赖无数据知识蒸馏优化生成器与集成模型,但难以应对数据异构问题,且在复杂数据集上因两阶段信息丢失(本地训练和知识回传)导致效果下降。本文提出FedSD2C框架,引入生成器直接从本地数据合成有意义的‘合成知识’(distillates),以替代不一致的本地模型进行通信,有效缓解数据异构影响并减少信息损失。实验证明,FedSD2C在复杂真实数据集上持续优于其他单轮联邦学习方法,性能最高达最佳基线的2.6倍。
原文摘要 · Abstract (English)
One-shot Federated learning (FL) is a powerful technology facilitating collaborative training of machine learning models in a single round of communication. While its superiority lies in communication efficiency and privacy preservation compared to iterative FL, one-shot FL often compromises model performance. Prior research has primarily focused on employing data-free knowledge distillation to optimize data generators and ensemble models for better aggregating local knowledge into the server model. However, these methods typically struggle with data heterogeneity, where inconsistent local data distributions can cause teachers to provide misleading knowledge. Additionally, they may encounter scalability issues with complex datasets due to inherent two-step information loss: first, during local training (from data to model), and second, when transferring knowledge to the server model (from model to inversed data). In this paper, we propose FedSD2C, a novel and practical one-shot FL framework designed to address these challenges. FedSD2C introduces a distiller to synthesize informative distillates directly from local data to reduce information loss and proposes sharing synthetic distillates instead of inconsistent local models to tackle data heterogeneity. Our empirical results demonstrate that FedSD2C consistently outperforms other one-shot FL methods with more complex and real datasets, achieving up to 2.6 the performance of the best baseline. Code: https://github.com/Carkham/FedSD2C
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。