arXiv:2605.12994cs.LG2026-05被引 1

提出隐私保护的矩阵动量优化方法,提升私有微调效果。

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

论文配图:DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum
图 1 · 摘自论文原文
  • 用矩阵动量+牛顿-舒尔正交化实现隐私保护训练。
  • 证明隐私保障不因后处理增加,且误差可分离分析。
  • 提出偏差修正版本,提升模型性能且不增加隐私开销。

我们研究基于Muon(一种使用动量后接牛顿-舒尔正交化的矩阵优化器)的差分隐私(DP)训练。尽管DP-SGD已被充分理解,但每样本裁剪、高斯噪声、动量与非线性正交化在Muon中的交互尚未系统分析。为此提出DP-Muon:对每样本矩阵梯度裁剪,对裁剪后平均添加高斯噪声,再进行动量和牛顿-舒尔正交化后处理。证明其隐私保障由对应同批子采样高斯会计仪保证,无额外隐私成本。优化方面,建立有限时间与渐近平稳性保证,误差分解为优化误差、裁剪残差、隐私噪声与牛顿-舒尔近似误差。发现隐私引入的偏差源于非线性牛顿-舒尔映射,而非线性动量缓冲区。由此提出偏纠正版本DP-MuonBC,消除主导输出偏差项,同时保持相同隐私保障。在E2E与DART上的实验表明,Muon风格矩阵更新提升私有微调效果,且DP-MuonBC进一步提高效用而无需增加隐私预算。

原文摘要 · Abstract (English)

We study differentially private (DP) training with Muon, a matrix-valued optimizer that updates hidden-layer weights using momentum followed by Newton--Schulz orthogonalization. While DP-SGD is well understood, the interaction between per-example clipping, Gaussian noise, momentum, and nonlinear orthogonalization in Muon has not been systematically analyzed. We formulate DP-Muon, a private Muon procedure that clips per-example matrix gradients, adds Gaussian noise to the clipped lot average, and then applies momentum and Newton--Schulz orthogonalization as post-processing. We prove that DP-Muon inherits the privacy guarantee certified by the corresponding same-lot subsampled Gaussian accountant, with no additional privacy cost from Muon-specific post-processing. On the optimization side, we establish finite-horizon and vanishing stationarity guarantees under per-matrix clipping, with bounds that separate optimization error, clipping residual, privacy noise, and Newton--Schulz approximation error. We further show that the DP-induced bias in Muon arises not in the linear momentum buffer itself, but after the nonlinear Newton--Schulz map, where Gaussian noise induces a matrix-valued heat-smoothing bias. This motivates DP-MuonBC, a bias-corrected variant that removes the leading output-level bias term while preserving the same privacy guarantee. Experiments on E2E and DART show that Muon-style matrix updates improve private fine-tuning, and that DP-MuonBC further improves utility without increasing the privacy budget.

差分隐私优化器矩阵更新私有微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。