用基因化压缩参数,让联邦学习通信量降4倍、初始化快9MB。
GENE-FL: Gene-Driven Parameter-Efficient Dynamic Federated Learning
- 通过高Fisher值参数筛选,提取可迁移的轻量知识片段。
- 通信成本降低4倍,动态客户端仅需9.04MB即可快速初始化。
- 适合数据异构性强、设备动态加入的实时联邦学习场景。
现实中的联邦学习系统常面临动态客户端与异构数据分布(DAFL)的挑战,导致通信效率低、模型初始化困难。受近期提出的Learngene范式启发,该范式将大模型压缩为跨任务的轻量级元信息片段,有效封装并传递核心知识,特别适用于需要高效通信与快速适应新数据分布的动态环境。本文提出一种基因驱动的参数高效动态联邦学习框架(GENE-FL)。首先,本地模型基于全局模型中高Fisher值的参数进行二次约束,这些参数被认为蕴含可泛化的通用知识;其次,利用参数敏感性分析对本地模型参数进行压缩,生成交互用的轻量级“learnGene”;最后,服务器将多个小型训练后的“learnGene”聚合为具备跨任务泛化能力的鲁棒“learnGene”,从而实现对动态异构客户端模型的快速初始化。大量实验表明,GENE-FL相比FEDAVG减少4倍通信开销,并仅需约9.04MB即可有效初始化异构客户端模型。
原文摘要 · Abstract (English)
Real-world \underline{F}ederated \underline{L}earning systems often encounter \underline{D}ynamic clients with \underline{A}gnostic and highly heterogeneous data distributions (DAFL), which pose challenges for efficient communication and model initialization. To address these challenges, we draw inspiration from the recently proposed Learngene paradigm, which compresses the large-scale model into lightweight, cross-task meta-information fragments. Learngene effectively encapsulates and communicates core knowledge, making it particularly well-suited for DAFL, where dynamic client participation requires communication efficiency and rapid adaptation to new data distributions. Based on this insight, we propose a Gene-driven parameter-efficient dynamic Federated Learning (GENE-FL) framework. First, local models perform quadratic constraints based on parameters with high Fisher values in the global model, as these parameters are considered to encapsulate generalizable knowledge. Second, we apply the strategy of parameter sensitivity analysis in local model parameters to condense the \textit{learnGene} for interaction. Finally, the server aggregates these small-scale trained \textit{learnGene}s into a robust \textit{learnGene} with cross-task generalization capability, facilitating the rapid initialization of dynamic agnostic client models. Extensive experimental results demonstrate that GENE-FL reduces \textbf{4 $\times$} communication costs compared to FEDAVG and effectively initializes agnostic client models with only about \textbf{9.04} MB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。