通过优化隐私损失变量,让私密训练模型更准更灵活。
PLRV-O: Advancing Differentially Private Deep Learning via Privacy Loss Random Variable Optimization
- 用可调噪声分布替代传统高斯噪声,实现隐私与效用解耦
- 在严格隐私下,图像分类准确率提升至94.03%,文本任务达92.20%
- 适合对隐私要求高且需适配不同训练配置的场景
差分隐私随机梯度下降(DP-SGD)是深度学习中保障隐私的标准方法,通常使用高斯机制对梯度更新加噪。然而,传统机制如高斯和拉普拉斯噪声仅由方差或尺度参数化,导致噪声大小直接关联隐私损失与效用下降,无法独立调控。当训练轮数T和批量大小B随任务变化时,这一问题加剧,微小噪声调整可能显著影响模型精度。为此,本文提出PLRV-O框架,构建广义参数化噪声分布空间,使隐私损失矩被精确刻画,并可独立优化以适应效用损失。该方法能系统性适配任务需求,包括模型规模、训练时长、采样策略及剪裁阈值,适用于训练与微调场景。实验表明,PLRV-O在严苛隐私约束下显著提升性能:在CIFAR-10上,微调的ViT达到94.03%准确率(ε≈0.5),优于高斯噪声的83.93%;在SST-2上,RoBERTa-large达到92.20%准确率(ε≈0.2),远超高斯噪声的50.25%。
原文摘要 · Abstract (English)
Differentially Private Stochastic Gradient Descent (DP-SGD) is a standard method for enforcing privacy in deep learning, typically using the Gaussian mechanism to perturb gradient updates. However, conventional mechanisms such as Gaussian and Laplacian noise are parameterized only by variance or scale. This single degree of freedom ties the magnitude of noise directly to both privacy loss and utility degradation, preventing independent control of these two factors. The problem becomes more pronounced when the number of composition rounds T and batch size B vary across tasks, as these variations induce task-dependent shifts in the privacy-utility trade-off, where small changes in noise parameters can disproportionately affect model accuracy. To address this limitation, we introduce PLRV-O, a framework that defines a broad search space of parameterized DP-SGD noise distributions, where privacy loss moments are tightly characterized yet can be optimized more independently with respect to utility loss. This formulation enables systematic adaptation of noise to task-specific requirements, including (i) model size, (ii) training duration, (iii) batch sampling strategies, and (iv) clipping thresholds under both training and fine-tuning settings. Empirical results demonstrate that PLRV-O substantially improves utility under strict privacy constraints. On CIFAR-10, a fine-tuned ViT achieves 94.03% accuracy at epsilon approximately 0.5, compared to 83.93% with Gaussian noise. On SST-2, RoBERTa-large reaches 92.20% accuracy at epsilon approximately 0.2, versus 50.25% with Gaussian.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。