针对隐私保护联邦学习中的优化难题，提出首个适配差分隐私的AdamW优化器。
DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models
- 通过稳定二阶矩方差、消除隐私噪声偏差，改进AdamW在隐私保护下的性能
- 在图像和语言模型上实现比现有最优方法高5.83%的准确率（Tiny-ImageNet）
- 适合研究隐私保护联邦学习或训练大规模模型的开发者使用
在联邦学习中平衡收敛效率与差分隐私下的鲁棒性是一大挑战。尽管AdamW能加速大规模模型训练，但直接应用于差分隐私联邦学习（DPFL）时存在三大问题：(i) 数据异质性与隐私噪声共同放大二阶矩估计方差，(ii) DP扰动导致二阶矩估计有偏，(iii) DP加剧AdamW对本地过拟合的敏感性，恶化客户端漂移。本文提出首个基于AdamW的差分隐私联邦学习优化器——DP-FedAdamW。它通过稳定二阶矩方差、消除DP引入的偏差、对齐本地更新与全局下降方向，恢复了AdamW在差分隐私下的有效性。理论上，我们建立了无偏的二阶矩估计器，证明了无需异质性假设的线性加速收敛率，并提供更紧的$(\varepsilon,δ)$-差分隐私保证。实验表明，DP-FedAdamW在语言与视觉Transformer及ResNet-18上均有效。在Tiny-ImageNet（Swin-Base，$\varepsilon=1$）上，性能优于当前最优方法5.83%。代码见附录。
原文摘要 · Abstract (English)
Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter $(\varepsilon,δ)$-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, $\varepsilon=1$), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83\%. The code is available in Appendix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。