差分隐私学习在长尾数据下表现差，新方法可提升罕见类准确率。
On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance
- 用带差分隐私的梯度下降优化长尾数据时，低频类难学。
- 改进的DP-AdamBC算法使罕见类别训练准确率提升约5%-8%。
- 适合关注隐私保护下长尾分布性能的研究者或工程师。
本文分析了常见私有学习优化算法在长尾类别不平衡分布下的优化行为。在简化模型中，采用差分隐私梯度下降（DP-GD）时，学习低频类别表现不佳；而估计二阶信息的优化算法则不受影响。特别地，移除损失曲率估计中差分隐私偏差的DP-AdamBC是避免长尾不平衡导致病态条件的关键，实验证明其在控制实验和真实数据上分别使最少见类别的训练准确率提高约8%和约5%。
原文摘要 · Abstract (English)
In this work, we analyze the optimization behaviour of common private learning optimization algorithms under heavy-tail class imbalanced distribution. We show that, in a stylized model, optimizing with Gradient Descent with differential privacy (DP-GD) suffers when learning low-frequency classes, whereas optimization algorithms that estimate second-order information do not. In particular, DP-AdamBC that removes the DP bias from estimating loss curvature is a crucial component to avoid the ill-condition caused by heavy-tail class imbalance, and empirically fits the data better with $\approx8\%$ and $\approx5\%$ increase in training accuracy when learning the least frequent classes on both controlled experiments and real data respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。