通过迭代遗忘与进化策略,提升联邦学习在非独立同分布数据下的泛化能力。
Forget to Generalize: Iterative Adaptation for Generalization in Federated Learning
- 每轮训练后随机或从深层重置部分参数,实现模型迭代演化
- 在非独立同分布数据下,平均准确率提升21.5%
- 可兼容任意联邦算法,适合隐私保护的分布式智能系统
网络环境天然异构,用户设备、地理位置、浏览行为和上下文导致数据高度多样且独特。联邦学习(FL)因能跨多样设备与服务实现隐私保护的协同机器学习而至关重要,但其在真实网络系统中普遍存在的非独立同分布(Non-IID)客户端分布下性能严重下降。本文提出一种新训练范式——迭代联邦适应(IFA),通过逐代遗忘与演化策略增强异构联邦环境下的泛化能力。具体而言,将训练划分为多代,在每代末尾选择一部分模型参数(随机或来自深层)进行重初始化。该迭代遗忘与演化机制使模型摆脱局部最优,保留全局相关表征。在CIFAR-10、MIT-Indoors和Stanford Dogs数据集上的大量实验表明,该方法显著提升全局准确率,尤其在跨客户端数据非独立同分布时表现优异。该方法可嵌入任意联邦算法以提升其泛化性能,平均提升达21.5%。本工作推动了面向真实异构分布式网络系统的可扩展隐私保护智能愿景。
原文摘要 · Abstract (English)
The Web is naturally heterogeneous with user devices, geographic regions, browsing patterns, and contexts all leading to highly diverse, unique datasets. Federated Learning (FL) is an important paradigm for the Web because it enables privacy-preserving, collaborative machine learning across diverse user devices, web services and clients without needing to centralize sensitive data. However, its performance degrades severely under non-IID client distributions that is prevalent in real-world web systems. In this work, we propose a new training paradigm - Iterative Federated Adaptation (IFA) - that enhances generalization in heterogeneous federated settings through generation-wise forget and evolve strategy. Specifically, we divide training into multiple generations and, at the end of each, select a fraction of model parameters (a) randomly or (b) from the later layers of the model and reinitialize them. This iterative forget and evolve schedule allows the model to escape local minima and preserve globally relevant representations. Extensive experiments on CIFAR-10, MIT-Indoors, and Stanford Dogs datasets show that the proposed approach improves global accuracy, especially when the data cross clients are Non-IID. This method can be implemented on top any federated algorithm to improve its generalization performance. We observe an average of 21.5%improvement across datasets. This work advances the vision of scalable, privacy-preserving intelligence for real-world heterogeneous and distributed web systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。