将压缩与隐私保护结合,用更少通信量实现更强隐私保护。
Sketched Gaussian Mechanism for Private Federated Learning
- 将随机投影压缩与高斯噪声直接结合,统一建模隐私保障。
- 相同噪声下,压缩维度越大,隐私保护越强,呈1/√b关系。
- 适用于梯度下降与自适应优化器,兼顾性能与隐私。
联邦学习中通信开销与隐私保护是两大挑战。为降低通信成本,常通过随机投影压缩客户端上传的模型更新;为保障隐私,普遍采用高斯机制(GM),即裁剪更新并添加高斯噪声以实现客户级差分隐私。现有研究分别分析压缩与GM的隐私效果,认为压缩提供由压缩维度决定的隐私,而GM需额外补充隐私。本文提出联合机制——Sketching Gaussian Mechanism(SGM),将压缩与高斯机制直接结合,并利用Rényi差分隐私工具进行联合隐私分析,结果比孤立分析更灵活、更紧致。特别地,我们证明在固定噪声强度下,SGM的隐私水平与$1/\sqrt{b}$成正比,其中$b$为压缩维度。这意味着在中等压缩维度下,相同噪声预算下SGM可显著优于传统GM。我们将SGM应用于基于梯度下降或自适应服务器优化器的联邦学习,建立了优化收敛性理论,其依赖参数量$d$仅呈对数关系。实验表明,在同等隐私水平下,基于SGM的联邦学习至少与非压缩私有方法相当,部分场景下表现更优。同时,使用自适应优化器进一步提升性能,且不牺牲隐私保证。
原文摘要 · Abstract (English)
Communication cost and privacy are two major considerations in federated learning (FL). For communication cost, gradient compression by sketching the clients' transmitted model updates is often used for reducing per-round communication. For privacy, the Gaussian mechanism (GM), which consists of clipping updates and adding Gaussian noise, is commonly used to guarantee client-level differential privacy. Existing literature on private FL analyzes privacy of sketching and GM in an isolated manner, illustrating that sketching provides privacy determined by the sketching dimension and that GM has to supply any additional desired privacy. In this paper, we introduce the Sketched Gaussian Mechanism (SGM), which directly combines sketching and the Gaussian mechanism for privacy. Using Rényi-DP tools, we present a joint analysis of SGM's overall privacy guarantee, which is significantly more flexible and sharper compared to isolated analysis of sketching and GM privacy. In particular, we prove that the privacy level of SGM for a fixed noise magnitude is proportional to $1/\sqrt{b}$, where $b$ is the sketching dimension, indicating that (for moderate $b$) SGM can provide much stronger privacy guarantees than the original GM under the same noise budget. We demonstrate the application of SGM to FL with either gradient descent or adaptive server optimizers, and establish theoretical results on optimization convergence, which exhibits only a logarithmic dependence on the number of parameters $d$. Experimental results confirm that at the same privacy level, SGM based FL is at least competitive with non-sketching private FL variants and outperforms them in some settings. Moreover, using adaptive optimization at the server improves empirical performance while maintaining the privacy guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。