减少更新参数坐标数也能保持私密训练效果，关键在选对坐标。
When Do Fewer Coordinates Suffice in DP-SGD?
- 分两阶段：先私有预热找重要坐标，再聚焦这些坐标训练。
- 当活跃维度较小时，噪声影响可降至活跃维度而非全维度。
- 适合低维活跃、无公开数据的隐私保护训练场景。
差分隐私随机梯度下降（DP-SGD）在每次更新中向所有参数坐标注入噪声，导致噪声能量随参数维度 $d$ 增大。本文探讨在不损失优化信号的前提下，能否仅更新较少坐标。提出 extsc{TP-TopK}（两阶段顶K DP-SGD），无需公共数据即可实现坐标稀疏的私密训练：先通过私有预热阶段识别关键坐标支持集，指导主训练阶段。给出一个条件判定坐标限制是否有益，理论证明在该条件下相关噪声项随活跃维度 $k$ 而非总维度 $d$ 缩放，并提供基于预热排序可靠性的下界。在 MNIST、FMNIST 和 CIFAR-10 上的实验表明，学习得到的坐标支持集比同等大小的随机支持集保留更多梯度能量，尤其当活跃维度小且预热评分有效时收益最大。
原文摘要 · Abstract (English)
Differentially private stochastic gradient descent (DP-SGD) injects noise into every updated coordinate, making the injected noise energy scale with the ambient parameter dimension \(d\). We ask when private training can update fewer coordinates without losing the signal needed for optimization. We propose \textsc{TP-TopK} (Two-Phase TopK DP-SGD), a two-phase method for coordinate-sparse private training without public data, in which a private warm-up phase identifies a coordinate support used to guide the main training phase. We give a criterion characterizing when coordinate restriction can be beneficial, show via a nonconvex stationarity bound that under this condition the relevant noise term scales with the active dimension \(k\) rather than the full parameter dimension \(d\), and provide a lower bound on the reliability of warm-up-based coordinate ranking. Experiments on MNIST, FMNIST, and CIFAR-10 show that learned coordinate supports can retain more gradient energy than size-matched random supports, with the largest gains when the active dimension is small and warm-up scores are informative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。