在联邦学习中安全地搜索最优超参数,保护隐私且不依赖具体任务。
DP-Hype: Federated Differentially Private Hyperparameter Search
- 通过客户端本地评估+联邦投票选择多数支持的超参数。
- 在小隐私预算下仍保持高搜索效率,证明可找到优质超参数。
- 适用于各类任务,特别适合注重隐私的医疗、金融场景。
联邦机器学习中的超参数调优显著影响模型性能。当在敏感数据上进行调优时,隐私成为关键挑战,差分隐私已成为可证明隐私的标准方案。标准联邦学习设定是客户端共同确定一组折衷的超参数(如学习率)。然而,现有隐私保护超参数调优方法多针对特定任务,未考虑聚合结果带来的隐私泄露,或隐私-效用权衡不佳。本文提出算法 DP-Hype,通过客户端本地评估并基于联邦投票机制进行隐私保护的超参数搜索。该方法选出多数客户端支持的折衷超参数,兼具可扩展性与任务无关性。我们证明了 DP-Hype 满足强差分隐私形式——客户端级差分隐私,且其隐私保证不随超参数数量增加而减弱。同时提供了效用边界,即找到优质超参数的概率,并将其实现为流行联邦框架 Flower 的子模块。我们在多个基准数据集的独立同分布及多种非独立同分布设置下进行了评估,结果表明即使在小隐私预算下,DP-Hype 仍具有高实用性。
原文摘要 · Abstract (English)
Tuning hyperparameters in federated machine learning can substantially impact model performance. When hyperparameters are tuned on sensitive data, privacy becomes an important challenge and to this end, differential privacy has emerged as the de facto standard for provable privacy. A standard setting in federated learning is that clients agree on a shared setup, i.e., find a compromise from a set of hyperparameters, like a model's learning rate. Yet, prior work on privacy-preserving hyperparameter tuning is tailored to specific learning tasks, does not account for the privacy leakage of aggregated results, or offers a sub-optimal privacy-utility trade-off. In this work, we present our algorithm DP-Hype, which performs a federated and privacy-preserving hyperparameter search by conducting a federated voting based on local hyperparameter evaluations of clients. In this way, DP-Hype selects hyperparameters that lead to a compromise supported by a majority of clients, while maintaining scalability and independence from specific learning tasks. We prove that DP-Hype preserves the strong notion of differential privacy called client-level differential privacy and, importantly, show that its privacy guarantees do not depend on the number of hyperparameters. We also provide bounds on its utility guarantees, that is, the probability of finding good hyperparameters, and implement DP-Hype as a submodule in the popular Flower framework for federated machine learning. In addition, we evaluate performance on multiple benchmark data sets in iid as well as multiple non-iid settings and demonstrate high utility of DP-Hype even under small privacy budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。