arXiv:2409.07734cs.DCcs.LG2024-09中稿 · ICDM2024 main conf…被引 7

无需数据集,用双生成器对抗蒸馏提升单轮联邦学习性能

DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learning

  • 训练两个互补的生成器,扩大本地模型的可探索空间
  • 在多个图像分类任务上,准确率显著超越现有最先进方法
  • 适合隐私敏感、无公共数据可用的单轮联邦学习场景

联邦学习(FL)是一种分布式机器学习范式,客户端通过共享模型信息而非私有数据集来协同训练全局模型。针对通信与隐私问题,单轮联邦学习(one-shot FL)已成为一种有前景的解决方案。然而,现有方法或依赖公开数据集,或仅适用于同质化模型设置,或仅从本地模型中蒸馏有限知识,难以训练出稳健的全局模型。为此,本文提出一种全新的无数据双生成器对抗蒸馏方法(DFDG),通过训练双生成器探索更广的本地模型训练空间。DFDG以对抗方式运行,包含双生成器训练与双模型蒸馏两部分:在双生成器训练中,分别优化生成器的保真度、迁移性与多样性,并引入交叉散度损失减少双生成器输出空间重叠;在双模型蒸馏阶段,训练好的双生成器协作生成训练数据以更新全局模型。大量实验表明,该方法在多个图像分类任务上相较现有最优基线实现了显著的准确率提升。

原文摘要 · Abstract (English)

Federated Learning (FL) is a distributed machine learning scheme in which clients jointly participate in the collaborative training of a global model by sharing model information rather than their private datasets. In light of concerns associated with communication and privacy, one-shot FL with a single communication round has emerged as a de facto promising solution. However, existing one-shot FL methods either require public datasets, focus on model homogeneous settings, or distill limited knowledge from local models, making it difficult or even impractical to train a robust global model. To address these limitations, we propose a new data-free dual-generator adversarial distillation method (namely DFDG) for one-shot FL, which can explore a broader local models' training space via training dual generators. DFDG is executed in an adversarial manner and comprises two parts: dual-generator training and dual-model distillation. In dual-generator training, we delve into each generator concerning fidelity, transferability and diversity to ensure its utility, and additionally tailor the cross-divergence loss to lessen the overlap of dual generators' output spaces. In dual-model distillation, the trained dual generators work together to provide the training data for updates of the global model. At last, our extensive experiments on various image classification tasks show that DFDG achieves significant performance gains in accuracy compared to SOTA baselines.

联邦学习无数据蒸馏对抗生成单轮学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。