揭示高维对抗训练中SGD的精确动态规律,解析学习率与正则化影响。
Homogenization of $\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent

- 基于ODE和SDE建模高维对抗训练的迭代过程,给出风险与最优解距离的确定性等价物。
- 发现恒定学习率无法保证对抗风险单调下降,且线搜索法不如Polyak步长有效。
- 揭示对抗训练等效于自适应学习率与正则化的标准最小二乘,适合研究理论机制者阅读。
我们构建了一个分析高维下单指标模型在高斯混合数据上进行ℓ₂-对抗训练的学习动态的框架,适用于流式随机梯度下降(SGD)。推导出广泛统计量(包括对抗风险和到对抗最优的距离)的确定性等价形式,其由常微分方程组(ODE)解决定。研究了两种理想化学习率调度:Polyak步长和精确线搜索。对于单类ℓ₂-对抗最小二乘,表明不同于无噪声的标准最小二乘,任意恒定学习率均不能保证SGD向对抗风险最小值单调收敛。识别出各向异性协方差和岭参数不匹配是导致精确线搜索性能劣于Polyak步长的主要原因。此外,引入一种称为对抗同质化SGD的随机微分方程(SDE),捕捉迭代统计量的演化。对于ℓ₂-对抗最小二乘,利用该SDE证明风险演化等价于具有自适应学习率和自适应ℓ₂-正则化的标准最小二乘的SGD。当动态收敛时,极限对抗风险和迭代点由一个不动点方程决定,极限迭代点等价于对应岭回归问题的解,其正则化参数为SGD的极限有效正则化参数。
原文摘要 · Abstract (English)
We develop a framework for analyzing the learning dynamics of $\ell_2$-adversarial training of single-index models on Gaussian mixtures in the high-dimensional limit under streaming stochastic gradient descent (SGD). We derive deterministic equivalents for a broad class of statistics of the SGD iterates, including the adversarial risk and distance to adversarial optimality, in terms of the solution to a system of ODEs. We use them to study two idealized learning rate schedules: the Polyak stepsize and exact line search. In the case of $\ell_2$-adversarial least squares with a single class, we show that, unlike noiseless standard least squares, no constant learning rate guarantees monotone descent of SGD towards a minimizer of the adversarial risk. We identify anisotropic covariance and a mismatch in ridge parameters as the main sources of suboptimality of exact line search relative to the Polyak stepsize. We also introduce a stochastic differential equation (SDE), called adversarial homogenized SGD, that captures the evolution of statistics of the iterates of SGD. For $\ell_2$-adversarial least squares, using this SDE, we show the evolution of the risk is equivalent, up to dimension-free constants, to that of SGD on standard least squares with an adaptive learning rate and adaptive $\ell_2$-regularization. When the dynamics converge, the limiting adversarial risk and SGD iterate are determined by a fixed-point equation, with the limiting iterate being equivalent to the solution of a ridge regression problem whose regularization parameter is the limiting effective regularization of SGD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。