arXiv:2411.01916cs.CV2024-11被引 4

用掩码自编码器实现高效联邦持续学习,解决遗忘与数据分布不均问题。

Masked Autoencoders are Parameter-Efficient Federated Continual Learners

  • 客户端用掩码自编码器重建图像,生成可上传的恢复信息
  • 服务器通过重建图像捕捉历史任务分布,提升分类性能
  • 适用于自监督预训练模型,特别适合资源受限场景

联邦学习是一种分布式学习范式,中央服务器聚合多个客户端本地模型的更新,从而在不上传私有数据的前提下完成学习,保障数据隐私。现有联邦学习方法主要针对静态数据设计,但现实应用中客户端需随时间学习新类别,这要求引入持续学习技术,形成联邦持续学习(FCL)。为应对灾难性遗忘和非独立同分布(non-IID)问题,我们提出使用掩码自编码器(MAE)作为参数高效的联邦持续学习者,命名为pMAE。pMAE在客户端通过图像重建学习重构提示(reconstructive prompt),在服务器端利用上传的恢复信息重建图像,捕捉过往任务和不同客户端的数据分布,并以此微调判别性提示和分类器参数,从而缓解全局层面的灾难性遗忘与non-IID问题。实验表明,pMAE性能可媲美现有基于提示的方法,且在使用自监督预训练变压器作为主干网络时能显著增强其效果。代码已开源:https://github.com/ycheoo/pMAE。

原文摘要 · Abstract (English)

Federated learning is a specific distributed learning paradigm in which a central server aggregates updates from multiple clients' local models, thereby enabling the server to learn without requiring clients to upload their private data, maintaining data privacy. While existing federated learning methods are primarily designed for static data, real-world applications often require clients to learn new categories over time. This challenge necessitates the integration of continual learning techniques, leading to federated continual learning (FCL). To address both catastrophic forgetting and non-IID issues, we propose to use masked autoencoders (MAEs) as parameter-efficient federated continual learners, called pMAE. pMAE learns reconstructive prompt on the client side through image reconstruction using MAE. On the server side, it reconstructs the uploaded restore information to capture the data distribution across previous tasks and different clients, using these reconstructed images to fine-tune discriminative prompt and classifier parameters tailored for classification, thereby alleviating catastrophic forgetting and non-IID issues on a global scale. Experimental results demonstrate that pMAE achieves performance comparable to existing prompt-based methods and can enhance their effectiveness, particularly when using self-supervised pre-trained transformers as the backbone. Code is available at: https://github.com/ycheoo/pMAE.

联邦学习持续学习自编码器参数高效

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。