解决联邦学习增量分类中的模型过载与遗忘问题
Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning

- 按样本动态路由至最优专家子集,实现知识获取与保留平衡
- 理论证明收敛速率达O(T⁻¹),实验优于当前最先进方法
- 适合数据异构、类别逐步增加的联邦学习场景
联邦学习(FL)作为一种分布式机器学习范式备受关注。然而,将其扩展到类别增量学习场景面临三大挑战:1)共享模型过载导致的能力冲突与灾难性遗忘;2)非独立同分布(Non-IID)数据带来的异质性;3)同步类对齐错位。本文提出一种新型框架——Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning(FedFMX),通过客户端间自适应专家专精化解上述难题。核心思想是将每个样本路由至一组联合优化知识获取与保留的专家。我们设计了基于费舍尔信息的专家评分(FRES)模块,通过费舍尔稳定性代价和梯度可塑性收益估计专家重要性;进一步引入自适应专家选择(AES)模块,量化边际贡献以动态确定专家子集;最后通过路由感知正则化(RAR)实现负载均衡与高效联邦训练。理论上证明了$/mathcal{O}(T^{-1})$的收敛速率。在多个基准数据集上的大量实验表明,FedFMX显著优于现有最先进方法。
原文摘要 · Abstract (English)
Federated Learning (FL) emerged as a promising distributed machine learning paradigm. However, extending FL to the class incremental learning scenarios introduces unique challenges: 1) Capacity conflict and catastrophic forgetting from the shared model overloading, 2) Heterogeneity from Non-Independent and Identically Distributed (Non-IID) data, and 3) Synchronized class misalignment. In this paper, we propose \textbf{F}isher-Routed \textbf{M}i\textbf{X}ture of Experts for \textbf{Fed}erated Class-Incremental Learning (\textsc{FedFMX}), a novel framework to address these challenges via adaptive expert specialization across clients. The crucial insight is to route each sample to an expert subset that jointly optimizes knowledge acquisition and retention. Specifically, we introduce a Fisher-Routed Expert Scoring (FRES) module to estimate expert importance via Fisher-based stability cost and gradient-based plasticity gain. Then, we design an Adaptive Expert Selection (AES) module by quantifying marginal contributions for adaptive expert subset determination. Finally, by the routing-aware regularization (RAR), we achieve load balance and efficient FL training. We theoretically prove the $\mathcal{O}(T^{-1})$ convergence rate. Extensive experiments on multiple benchmarks compared with state-of-the-art methods demonstrate the superiority of \textsc{FedFMX}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。