用信息论框架分析联邦学习的泛化能力,揭示隐私与泛化的关系。
Generalization in Federated Learning: A Conditional Mutual Information Framework
- 基于条件互信息构建两级泛化分析框架,引入超客户机概念。
- 导出多个非空泛化界,小经验风险下达到最优收敛速率。
- 适用于不同聚合策略,对隐私保护型学习有指导意义。
联邦学习(FL)是一种广泛采用的隐私保护分布式学习框架,但其泛化性能相较于集中式学习仍较少被研究。在FL中,泛化误差包含两个部分:样本外差距(衡量参与客户端的经验风险与真实风险之差),以及参与差距(衡量参与与未参与客户端的风险差异)。本文通过条件互信息(CMI)框架进行信息论分析,研究FL的两级泛化。不同于传统的超样本CMI框架,我们提出超客户机构造以适配FL的两级泛化设置。推导出多个基于CMI的界,包括假设相关的CMI界,阐明了FL中的隐私约束如何隐含泛化保证。此外,我们提出快速率评估的CMI界,在小经验风险情形下恢复了两级FL泛化最优已知收敛率。针对特定的模型聚合策略和结构化损失函数,进一步优化边界,提升关于参与客户端数量的收敛速率。实验验证表明,所提出的评估CMI界是非空的,能准确捕捉FL算法的泛化行为。
原文摘要 · Abstract (English)
Federated learning (FL) is a widely adopted privacy-preserving distributed learning framework, yet its generalization performance remains less explored compared to centralized learning. In FL, the generalization error consists of two components: the out-of-sample gap, which measures the gap between the empirical and true risk for participating clients, and the participation gap, which quantifies the risk difference between participating and non-participating clients. In this work, we apply an information-theoretic analysis via the conditional mutual information (CMI) framework to study FL's two-level generalization. Beyond the traditional supersample-based CMI framework, we introduce a superclient construction to accommodate the two-level generalization setting in FL. We derive multiple CMI-based bounds, including hypothesis-based CMI bounds, illustrating how privacy constraints in FL can imply generalization guarantees. Furthermore, we propose fast-rate evaluated CMI bounds that recover the best-known convergence rate for two-level FL generalization in the small empirical risk regime. For specific FL model aggregation strategies and structured loss functions, we refine our bounds to achieve improved convergence rates with respect to the number of participating clients. Empirical evaluations confirm that our evaluated CMI bounds are non-vacuous and accurately capture the generalization behavior of FL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。