提出精准遗忘机制,让联邦持续学习更好应对数据异质性。
Accurate Forgetting for Heterogeneous Federated Continual Learning
- 设计生成回放方法,用概率模型判断旧知识可信度,选择性遗忘。
- 在异构数据下显著降低偏差,准确率提升5.2%以上。
- 适合数据差异大、任务不相关的联邦学习场景使用。
近年来,联邦学习(FL)受到广泛关注,但客户端在序列学习中的情境仍研究不足。将联邦学习与持续学习结合形成联邦持续学习(FCL),面临严峻挑战:客户端间数据/任务可能无关甚至对立。在联邦场景中,客户端间的统计异质性和数据噪声会产生虚假相关性,导致特征学习偏差。现有持续学习策略强调完全利用历史知识,但我们发现,在此设定下,主动遗忘偏差信息反而有益。为此,本文提出‘精准遗忘’(Accurate Forgetting, AF)新概念,并设计一种新型生成回放方法~\method~,在联邦网络中选择性地利用历史知识。该方法基于归一化流模型构建概率框架,量化先前知识的可信度。大量实验验证了该方法优于基线模型,在多个异构数据集上平均提升5.2%以上准确率。
原文摘要 · Abstract (English)
Recent years have witnessed a burgeoning interest in federated learning (FL). However, the contexts in which clients engage in sequential learning remain under-explored. Bridging FL and continual learning (CL) gives rise to a challenging practical problem: federated continual learning (FCL). Existing research in FCL primarily focuses on mitigating the catastrophic forgetting issue of continual learning while collaborating with other clients. We argue that the forgetting phenomena are not invariably detrimental. In this paper, we consider a more practical and challenging FCL setting characterized by potentially unrelated or even antagonistic data/tasks across different clients. In the FL scenario, statistical heterogeneity and data noise among clients may exhibit spurious correlations which result in biased feature learning. While existing CL strategies focus on a complete utilization of previous knowledge, we found that forgetting biased information is beneficial in our study. Therefore, we propose a new concept accurate forgetting (AF) and develop a novel generative-replay method~\method~which selectively utilizes previous knowledge in federated networks. We employ a probabilistic framework based on a normalizing flow model to quantify the credibility of previous knowledge. Comprehensive experiments affirm the superiority of our method over baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。