通过几何感知裁剪提升私有训练效果,让梯度更新更精准。
GeoClip: Geometry-Aware Clipping for Differentially Private SGD
- 在梯度分布几何结构上进行裁剪和扰动,而非标准坐标系
- 自适应估计变换矩阵,不增加额外隐私开销
- 在表格与图像数据上均优于现有方法,相同隐私下性能更好
差分隐私随机梯度下降(DP-SGD)是实现可证明隐私保障的主流训练方法。其关键挑战在于设置每样本梯度裁剪阈值,该阈值显著影响隐私与效用之间的权衡。尽管近期自适应方法通过训练过程中调整阈值提升了性能,但它们仍在标准坐标系中操作,未能考虑梯度各分量间的相关性。我们提出GeoClip,一种几何感知框架,将梯度裁剪与扰动置于与梯度分布几何结构对齐的变换基下。GeoClip通过仅使用先前发布的含噪梯度自适应估计该变换,且不引入额外隐私成本。我们为GeoClip提供了收敛性保证,并推导出最小化噪声添加量的同时控制梯度裁剪概率的最优变换闭式解。在表格与图像数据集上的实验表明,GeoClip在相同隐私预算下始终优于现有自适应裁剪方法。
原文摘要 · Abstract (English)
Differentially private stochastic gradient descent (DP-SGD) is the most widely used method for training machine learning models with provable privacy guarantees. A key challenge in DP-SGD is setting the per-sample gradient clipping threshold, which significantly affects the trade-off between privacy and utility. While recent adaptive methods improve performance by adjusting this threshold during training, they operate in the standard coordinate system and fail to account for correlations across the coordinates of the gradient. We propose GeoClip, a geometry-aware framework that clips and perturbs gradients in a transformed basis aligned with the geometry of the gradient distribution. GeoClip adaptively estimates this transformation using only previously released noisy gradients, incurring no additional privacy cost. We provide convergence guarantees for GeoClip and derive a closed-form solution for the optimal transformation that minimizes the amount of noise added while keeping the probability of gradient clipping under control. Experiments on both tabular and image datasets demonstrate that GeoClip consistently outperforms existing adaptive clipping methods under the same privacy budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。