arXiv:2409.13136cs.LGcs.CR2024-09中稿 · ACM MM 2023被引 36

解决联邦学习中标签分布不均问题,提升模型性能。

Federated Learning with Label-Masking Distillation

  • 按标签数量分多数/少数标签,分层蒸馏知识
  • 通过掩码机制增强少数类学习,准确率显著提升
  • 轻量版无需额外教师模型,适合资源受限设备

联邦学习通过全局服务器协调多个本地客户端的数据,实现隐私保护下的协同建模。本文关注客户端间标签分布偏斜问题——由于用户行为差异,各客户端的标签分布显著不同。现有方法因未能有效利用标签分布信息,常导致次优优化。为此,提出标签掩码蒸馏方法FedLMD,训练时将标签分为多数与少数类别:客户端模型从本地数据学习多数类知识;蒸馏过程中屏蔽全局模型对多数类的预测,使其聚焦于保留客户端的少数类知识。大量实验表明,该方法在多种场景下达到领先性能。此外,针对客户端资源有限的问题,提出轻量变体FedLMD-Tf,无需额外教师模型,在不增加计算开销的前提下超越现有轻量级方法。代码已开源。

原文摘要 · Abstract (English)

Federated learning provides a privacy-preserving manner to collaboratively train models on data distributed over multiple local clients via the coordination of a global server. In this paper, we focus on label distribution skew in federated learning, where due to the different user behavior of the client, label distributions between different clients are significantly different. When faced with such cases, most existing methods will lead to a suboptimal optimization due to the inadequate utilization of label distribution information in clients. Inspired by this, we propose a label-masking distillation approach termed FedLMD to facilitate federated learning via perceiving the various label distributions of each client. We classify the labels into majority and minority labels based on the number of examples per class during training. The client model learns the knowledge of majority labels from local data. The process of distillation masks out the predictions of majority labels from the global model, so that it can focus more on preserving the minority label knowledge of the client. A series of experiments show that the proposed approach can achieve state-of-the-art performance in various cases. Moreover, considering the limited resources of the clients, we propose a variant FedLMD-Tf that does not require an additional teacher, which outperforms previous lightweight approaches without increasing computational costs. Our code is available at https://github.com/wnma3mz/FedLMD.

联邦学习标签偏斜知识蒸馏轻量模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。