用信息论分析联邦学习中的概念漂移,提出自适应优化算法。
An Information-Theoretic Analysis for Federated Learning under Concept Drift
- 将概念漂移建模为马尔可夫链,引入平稳泛化误差衡量未来数据适应能力。
- 三种漂移模式下,新方法在性能上显著优于现有方法,提升长期稳定性。
- 适合关注动态数据场景下的联邦学习系统设计与优化的研究者。
联邦学习(FL)通常在静态数据集上训练模型,但现实世界数据常以流形式持续到来,分布不断变化,导致性能下降,即概念漂移。本文基于信息论分析概念漂移对联邦学习的影响,提出一种缓解性能下降的算法。将概念漂移建模为马尔可夫链,引入“平稳泛化误差”评估模型对未来未见数据的适应能力,其上界通过KL散度和互信息推导得出。研究了周期性、渐进性和随机性三种漂移模式对联邦学习性能的影响。受此启发,提出一种正则化经验风险最小化的算法,引入KL散度和互信息以增强长期性能,并通过识别帕累托前沿探索性能-成本权衡。使用Raspberry Pi4构建联邦学习测试平台进行验证,实验结果与理论分析一致,证实漂移模式显著影响性能,所提方法在三类漂移模式下均优于现有方法,证明其在应对概念漂移方面的有效性。
原文摘要 · Abstract (English)
Recent studies in federated learning (FL) commonly train models on static datasets. However, real-world data often arrives as streams with shifting distributions, causing performance degradation known as concept drift. This paper analyzes FL performance under concept drift using information theory and proposes an algorithm to mitigate the performance degradation. We model concept drift as a Markov chain and introduce the \emph{Stationary Generalization Error} to assess a model's capability to capture characteristics of future unseen data. Its upper bound is derived using KL divergence and mutual information. We study three drift patterns (periodic, gradual, and random) and their impact on FL performance. Inspired by this, we propose an algorithm that regularizes the empirical risk minimization approach with KL divergence and mutual information, thereby enhancing long-term performance. We also explore the performance-cost tradeoff by identifying a Pareto front. To validate our approach, we build an FL testbed using Raspberry Pi4 devices. Experimental results corroborate with theoretical findings, confirming that drift patterns significantly affect performance. Our method consistently outperforms existing approaches for these three patterns, demonstrating its effectiveness in adapting concept drift in FL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。