提出无需调参的差分隐私优化方法，兼顾高效与隐私保护。
Towards hyperparameter-free optimization with differential privacy
- 用自动学习率调度替代人工调参，适配各类模型与优化器。
- 结合逐样本梯度裁剪，实现接近非私有优化的计算效率。
- 在多语言和视觉任务中达当前最优差分隐私性能，适合实际部署。
差分隐私（DP）是一种保护深度学习训练数据的隐私范式。模型性能高度依赖于训练超参数，尤其是学习率调度，因此需对数据进行精细调参。实践中常通过网格搜索调整学习率，这不仅计算开销大（需多次运行），还因超参数选择依赖数据而增加数据泄露风险。本文将自动学习率调度应用于任意模型与优化器的差分隐私优化，显著降低甚至消除调参成本。结合自动逐样本梯度裁剪，该方法在计算效率上几乎等同于标准非私有优化，并在多种语言与视觉任务中达到当前最优的差分隐私性能。
原文摘要 · Abstract (English)
Differential privacy (DP) is a privacy-preserving paradigm that protects the training data when training deep learning models. Critically, the performance of models is determined by the training hyperparameters, especially those of the learning rate schedule, thus requiring fine-grained hyperparameter tuning on the data. In practice, it is common to tune the learning rate hyperparameters through the grid search that (1) is computationally expensive as multiple runs are needed, and (2) increases the risk of data leakage as the selection of hyperparameters is data-dependent. In this work, we adapt the automatic learning rate schedule to DP optimization for any models and optimizers, so as to significantly mitigate or even eliminate the cost of hyperparameter tuning when applied together with automatic per-sample gradient clipping. Our hyperparameter-free DP optimization is almost as computationally efficient as the standard non-DP optimization, and achieves state-of-the-art DP performance on various language and vision tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。