发现隐私保护微调中梯度分布变化导致最优超参与理论预期相反。
On Optimal Hyperparameters for Differentially Private Deep Transfer Learning
- 通过分析梯度分布，揭示隐私保护微调中超参选择的机制差异。
- 在固定训练轮次下，大批次反而优于小批次，与传统直觉相反。
- 不同任务间使用同一组超参会严重降低性能，尤其在隐私或算力变化时。
差分隐私（DP）迁移学习——即在私有数据上微调预训练模型——是当前在隐私约束下训练大型模型的前沿方法。本文聚焦两个关键超参数：裁剪阈值 $C$ 和批量大小 $B$。我们发现，现有理论认为更强隐私需更小 $C$，但实证结果却显示在强隐私条件下更大的 $C$ 表现更好，这一矛盾源于梯度分布的变化。在固定训练轮次（有限算力预算）的前提下，现有 $B$ 的调参启发式方法失效，而累积的差分隐私噪声能更好解释为何大或小批次更优。此外，跨任务统一使用单一 $(C,B)$ 设置会导致性能下降，尤其是在隐私强度或计算资源从宽松变为严格时，这可通过将裁剪视为梯度重加权、并分析累积噪声来解释。
原文摘要 · Abstract (English)
Differentially private (DP) transfer learning, i.e., fine-tuning a pretrained model on private data, is the current state-of-the-art approach for training large models under privacy constraints. We focus on two key hyperparameters in this setting: the clipping bound $C$ and batch size $B$. We show a clear mismatch between the current theoretical understanding of how to choose an optimal $C$ (stronger privacy requires smaller $C$) and empirical outcomes (larger $C$ performs better under strong privacy), caused by changes in the gradient distributions. Assuming a limited compute budget (fixed epochs), we demonstrate that the existing heuristics for tuning $B$ do not work, while cumulative DP noise better explains whether smaller or larger batches perform better. We also highlight how the common practice of using a single $(C,B)$ setting across tasks can lead to suboptimal performance. We find that performance drops especially when moving between loose and tight privacy and between plentiful and limited compute, which we explain by analyzing clipping as a form of gradient re-weighting and examining cumulative DP noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。