提出轻量级自适应私有优化器，提升隐私训练效率与性能。
DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuning
- 基于稀疏性设计的内存高效自适应优化方法
- 在多个基准上优于现有私有优化器，达最优收敛速率
- 适合需要高隐私保障的模型微调与大规模训练
自适应优化器是非隐私训练中的标准选择，常带来更快收敛和更好性能。相比之下，差分隐私（DP）训练仍主要依赖DP-SGD，通常需大量计算和超参数调优。我们提出DP-MicroAdam，一种内存高效且具备稀疏感知能力的自适应差分隐私优化器。理论上证明，DP-MicroAdam在随机非凸优化中可达到最优的$/mathcal{O}(1/ ext{sqrt}{T})$收敛率，仅受隐私相关常数影响。实验表明，该方法在多个基准上超越现有自适应私有优化器，其准确率在CIFAR-10、大规模ImageNet训练以及预训练Transformer的私有微调任务中均达到或超过DP-SGD水平。结果表明，自适应优化可在差分隐私下同时提升性能与稳定性。
原文摘要 · Abstract (English)
Adaptive optimizers are the de facto standard in non-private training as they often enable faster convergence and improved performance. In contrast, differentially private (DP) training is still predominantly performed with DP-SGD, typically requiring extensive compute and hyperparameter tuning. We propose DP-MicroAdam, a memory-efficient and sparsity-aware adaptive DP optimizer. We prove that DP-MicroAdam converges in stochastic non-convex optimization at the optimal $\mathcal{O}(1/\sqrt{T})$ rate, up to privacy-dependent constants. Empirically, DP-MicroAdam outperforms existing adaptive DP optimizers and achieves competitive or superior accuracy compared to DP-SGD across a range of benchmarks, including CIFAR-10, large-scale ImageNet training, and private fine-tuning of pretrained transformers. These results demonstrate that adaptive optimization can improve both performance and stability under differential privacy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。