arXiv:2605.03425cs.LG2026-05被引 1

提出新私有优化器FiBeR,解决滤波噪声干扰下的偏差问题。

FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction

论文配图:FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction
图 1 · 摘自论文原文
  • 在创新空间滤波降噪,重构梯度估计
  • 引入滤波感知的二阶矩校准,提升训练稳定性
  • 适合需要高隐私保护的视觉与语言任务

差分隐私(DP)训练通过向梯度添加噪声来保护个体样本,但注入的噪声会与自适应优化器产生非平凡交互。现有方法对私有化梯度进行时间滤波以降低方差,但滤波会改变AdamW二阶动量累积器所见的DP噪声统计特性。因此,针对未滤波DP噪声设计的偏差校正(如减去sigma_w平方)在滤波存在时可能失准。本文提出FiBeR,一种专为时间滤波后的私有化梯度设计的DP优化器:(i) 在创新空间中通过滤波残差流并积分形成滤波梯度估计;(ii) 将两点观测几何与创新增益解耦,实现独立调参;(iii) 引入滤波感知的二阶矩校准,减去衰减后的DP噪声贡献A(omega) sigma_w²,其中A(omega)对创新滤波有闭式表达,可推广至一般稳定线性滤波器。在视觉与语言基准上,FiBeR持续显著提升DP优化器性能,在多个任务上超越同等隐私约束下的现有最优结果。

原文摘要 · Abstract (English)

Differentially private (DP) training protects individual examples by adding noise to gradients, but the injected noise interacts nontrivially with adaptive optimizers. Recent DP methods temporally filter privatized gradients to reduce variance; however, filtering also changes the DP noise statistics seen by AdamW's second-moment accumulator. As a result, bias corrections derived for unfiltered DP noise, such as subtracting sigma_w squared, can become miscalibrated when filtering is present. We propose FiBeR, a DP optimizer designed for temporally filtered privatized gradients. FiBeR (i) performs denoising in innovation space by filtering the residual stream and integrating it to form the filtered gradient estimate, (ii) decouples the two-point observation geometry from the innovation gain to enable independent tuning, and (iii) introduces a filter-aware second-moment calibration that subtracts the attenuated DP noise contribution A(omega) sigma_w squared, where A(omega) is derived in closed form for the innovation filter and can be computed for general stable linear filters. Across vision and language benchmarks, FiBeR consistently demonstrates substantial improvements in the performance of DP optimizers, surpassing state-of-the-art results under equivalent privacy constraints on multiple tasks.

差分隐私优化器机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。