无需缓存数据,用新方法解决联邦学习中遗忘问题。
Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence
- 提出FedSSI算法,改进神经网络权重更新规则。
- 在数据分布不均时仍保持高准确率,比现有方法提升5.2%。
- 适合资源受限且注重隐私的分布式设备场景。
持续联邦学习(CFL)使分布式设备能协作学习不断变化的新任务,同时避免对旧任务的知识遗忘。现有方法多依赖大量历史数据重放,但会占用内存并可能泄露隐私。为此,本文探索不需数据缓存的正则化方法。实验发现,传统正则化如突触智能(Synaptic Intelligence)在数据同质时表现良好,但在异构数据下失效。针对此问题,提出新型正则化算法FedSSI,专为异构数据下的CFL设计。该方法无需重放数据,显著降低计算开销,并有效缓解数据异构带来的性能下降。大量实验表明,FedSSI在多个数据集上均优于当前最优方法,平均提升5.2%。
原文摘要 · Abstract (English)
Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。