从镜面下降视角解析平滑符号下降的优化机制，揭示稳定常数调优对解精度的影响。
A Mirror Descent Perspective of Smoothed Sign Descent
- 构建新镜面映射，使平滑符号下降等价于低维对偶动态。
- 收敛解为带布雷格曼散度函数的近似KKT点，ε越小误差越低。
- 适合研究优化算法隐式正则化机制的学者参考。
Woodworth 等人（2020）表明，过参数化问题中梯度下降的优化动态可视为由镜面映射诱导的低维对偶动态，从镜面下降视角解释了隐式正则化现象。然而，该方法不适用于更新方向偏离真实梯度的算法（如 ADAM）。本文利用镜面下降框架研究回归问题中带有稳定常数 ε 的平滑符号下降的动力学。在一定假设下，提出一个镜面映射，使其与对偶动态等价。通过分析对偶动态，将收敛解表征为最小化布雷格曼散度风格函数的近似 KKT 点，并证明调节稳定常数 ε 可降低 KKT 误差。
原文摘要 · Abstract (English)
Recent work by Woodworth et al. (2020) shows that the optimization dynamics of gradient descent for overparameterized problems can be viewed as low-dimensional dual dynamics induced by a mirror map, explaining the implicit regularization phenomenon from the mirror descent perspective. However, the methodology does not apply to algorithms where update directions deviate from true gradients, such as ADAM. We use the mirror descent framework to study the dynamics of smoothed sign descent with a stability constant $\varepsilon$ for regression problems. We propose a mirror map that establishes equivalence to dual dynamics under some assumptions. By studying dual dynamics, we characterize the convergent solution as an approximate KKT point of minimizing a Bregman divergence style function, and show the benefit of tuning the stability constant $\varepsilon$ to reduce the KKT error.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。