解决联邦学习增量更新时的遗忘问题,仅用一次通信完成新任务学习。
Catastrophic Forgetting Resilient One-Shot Incremental Federated Learning
- 通过客户端发送类别嵌入,服务器用扩散模型生成新数据进行训练。
- 在三个数据集上优于传统和一次性联邦学习方法,分类准确率提升显著。
- 适合隐私敏感、通信受限的实时增量学习场景,如医疗或边缘设备。
现代大数据系统产生大量异构且地理分散的数据流,规模大且具有隐私敏感性,使得集中化处理困难。虽然联邦学习(FL)提供了保护隐私的训练机制,但其假设数据流静态,需多轮协作训练,难以应对增量数据。本文提出一次性增量联邦学习(OSI-FL),首个同时应对通信开销与灾难性遗忘的框架。客户端利用冻结的视觉语言模型(VLM)生成类别特定嵌入,在单轮通信中传至服务器,由预训练扩散模型据此合成与客户端数据分布相似的新样本,用于服务器端训练。然而仍存在两个挑战:一是增量任务需重新训练全局模型;二是后续任务引入灾难性遗忘。为此,我们引入选择性样本保留(SSR),基于样本损失识别并保留每类-任务组合中前p%最具信息量的样本,确保代表性样本在后续迭代中持续参与训练,从而抑制遗忘。实验表明,OSI-FL在三个基准数据集上的类别增量和领域增量场景中均超越基线,包括传统与一次性联邦学习方法。
原文摘要 · Abstract (English)
Modern big-data systems generate massive, heterogeneous, and geographically dispersed streams that are large-scale and privacy-sensitive, making centralization challenging. While federated learning (FL) provides a privacy-enhancing training mechanism, it assumes a static data flow and learns a collaborative model over multiple rounds, making learning with \textit{incremental} data challenging in limited-communication scenarios. This paper presents One-Shot Incremental Federated Learning (OSI-FL), the first FL framework that addresses the dual challenges of communication overhead and catastrophic forgetting. OSI-FL communicates category-specific embeddings, devised by a frozen vision-language model (VLM) from each client in a single communication round, which a pre-trained diffusion model at the server uses to synthesize new data similar to the client's data distribution. The synthesized samples are used on the server for training. However, two challenges still persist: i) tasks arriving incrementally need to retrain the global model, and ii) as future tasks arrive, retraining the model introduces catastrophic forgetting. To this end, we augment training with Selective Sample Retention (SSR), which identifies and retains the top-p most informative samples per category and task pair based on sample loss. SSR bounds forgetting by ensuring that representative retained samples are incorporated into training in further iterations. The experimental results indicate that OSI-FL outperforms baselines, including traditional and one-shot FL approaches, in both class-incremental and domain-incremental scenarios across three benchmark datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。