提出新型采样方法，兼顾训练效率与隐私保护
Balls-and-Bins Sampling for DP-SGD
- 采用球桶采样模拟洗牌，实现高效训练
- 相同噪声倍数下，模型性能接近洗牌方法
- 隐私增益优于泊松采样，适合实际应用
本文提出球桶采样（Balls-and-Bins sampling）用于差分隐私优化方法，如DP-SGD。尽管实践中普遍采用洗牌机制，但隐私会计通常假设使用泊松采样。近期研究指出，基于洗牌的DP-SGD在实际参数设置下隐私成本显著更高。本工作表明，球桶采样兼具两者优势：其实施方式类似洗牌，且在相同噪声倍数下，模型性能与洗牌方法相当；同时，在实际参数范围内，其隐私增益优于或等同于泊松采样。
原文摘要 · Abstract (English)
We introduce the Balls-and-Bins sampling for differentially private (DP) optimization methods such as DP-SGD. While it has been common practice to use some form of shuffling in DP-SGD implementations, privacy accounting algorithms have typically assumed that Poisson subsampling is used instead. Recent work by Chua et al. (ICML 2024), however, pointed out that shuffling based DP-SGD can have a much larger privacy cost in practical regimes of parameters. In this work we show that the Balls-and-Bins sampling achieves the "best-of-both" samplers, namely, the implementation of Balls-and-Bins sampling is similar to that of Shuffling and models trained using DP-SGD with Balls-and-Bins sampling achieve utility comparable to those trained using DP-SGD with Shuffling at the same noise multiplier, and yet, Balls-and-Bins sampling enjoys similar-or-better privacy amplification as compared to Poisson subsampling in practical regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。