改进差分隐私训练的采样方法,让每条数据参与次数更均衡,提升隐私保护效果。
Less Random, More Private: What is the Optimal Subsampling Scheme for DP-SGD?
- 采用固定参与次数的结构化采样策略,减少数据参与波动性
- 在低噪声场景下,相比传统随机采样可降低9.6%的噪声需求
- 适用于高精度隐私保护训练,尤其适合追求实用性的研究者
在差分隐私机器学习中,泊松采样因易于分析隐私放大而被广泛使用,但其随机性导致样本参与训练次数差异大,造成显著的参与方差。本文证明这种方差不仅是实践问题,更是隐私放大效率低下的根本原因。我们提出平衡迭代采样(BIS),使每个样本恰好参与固定次数的训练迭代,理论上在噪声趋近0和无穷大时均达到最优隐私放大。分析表明,隐私与噪声的权衡不依赖最大化随机性,而是通过消除参与方差并保持每轮参与均匀性来实现。为实现有限噪声下的精确保证,我们设计了针对BIS的近似精确蒙特卡洛会计工具,消除了现有RDP和组合式PLD分析的理论松弛。在60多个实际配置上的评估显示,BIS在低噪声场景中持续优于泊松采样,噪声乘数最高可降低9.6%。结果推翻了‘更多采样随机性带来更强隐私’的普遍直觉:在DP-SGD中,结构化参与反而更高效、更私密。代码已开源。
原文摘要 · Abstract (English)
Poisson subsampling is the default sampling scheme in differentially private machine learning, largely because its unstructured randomness yields tractable privacy amplification analyses. Yet this same randomness introduces substantial participation variance: each sample appears in very different numbers of training iterations. In this work, we show that this variance is not merely a practical artifact to be tolerated, but a fundamental source of suboptimal privacy amplification. We prove that Balanced Iteration Subsampling (BIS), a structured scheme in which each sample participates in exactly a fixed number of iterations, achieves stronger privacy amplification than Poisson subsampling and is optimal at both extremes of the noise spectrum ($σ\to 0$ and $σ\to \infty$). Our analysis reveals that the privacy-noise tradeoff is governed not by maximizing randomness, but by eliminating participation variance while preserving uniform marginal participation across iterations. To translate this asymptotic theory into finite-noise guarantees, we introduce a practical near-exact Monte Carlo accountant for BIS, which removes the analytical slack of existing RDP and composition-based PLD analyses. Evaluations across more than 60 practical DP-SGD configurations show that BIS consistently outperforms Poisson subsampling in the low-noise regimes most relevant for high-utility private training, reducing the required noise multiplier by up to $9.6\%$. These results overturn the common intuition that more sampling randomness necessarily yields stronger privacy amplification: in DP-SGD, structured participation can be both more practical and more private. Our implementation is available at https://github.com/dong-xin-ao-andy/bis-mc-accountant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。