arXiv:2602.10584cs.LG2026-02被引 2

用模型参数设计动态梯度裁剪,提升隐私训练稳定性。

When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning

  • 基于权重矩阵谱分解的轻量诊断,实时估计训练稳定性
  • 通过反馈控制器在对数域乘法更新裁剪阈值
  • 无需额外计算开销,且不增加隐私损耗

敏感数据的隐私保护训练通常依赖差分隐私随机优化,结合梯度裁剪与高斯噪声。裁剪阈值是关键控制参数:过小会导致系统性过度裁剪引发优化偏差;过大则噪声主导更新,降低精度。现有自适应裁剪方法常依赖每样本梯度范数统计,增加计算开销并受数据集和架构影响。我们提出一种控制驱动的裁剪策略,通过周期性探查步骤中仅由模型参数生成的轻量级谱诊断,分析指定权重矩阵的谱分解,估计与训练稳定性相关的重尾谱指标。该指标随时间平滑后输入有界反馈控制器,在对数域中以乘法方式更新裁剪阈值。由于控制器仅使用隐私训练过程中产生的参数,阈值更新属于后处理,不会在标准组合计账下增加隐私损失。

原文摘要 · Abstract (English)

Privacy-preserving training on sensitive data commonly relies on differentially private stochastic optimization with gradient clipping and Gaussian noise. The clipping threshold is a critical control knob: if set too small, systematic over-clipping induces optimization bias; if too large, injected noise dominates updates and degrades accuracy. Existing adaptive clipping methods often depend on per-example gradient norm statistics, adding computational overhead and introducing sensitivity to datasets and architectures. We propose a control-driven clipping strategy that adapts the threshold using a lightweight, weight-only spectral diagnostic computed from model parameters. At periodic probe steps, the method analyzes a designated weight matrix via spectral decomposition and estimates a heavy-tailed spectral indicator associated with training stability. This indicator is smoothed over time and fed into a bounded feedback controller that updates the clipping threshold multiplicatively in the log domain. Because the controller uses only parameters produced during privacy-preserving training, the resulting threshold updates are post-processing and do not increase privacy loss beyond that of the underlying DP optimizer under standard composition accounting.

差分隐私梯度裁剪控制理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。