非欧几何下的镜像下降对初始值极度敏感,可能指数级放大微小误差。
Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity
- 引入非二次正则化后,镜像下降对初始扰动的敏感性远超梯度下降
- 三维构造中,ε扰动经T步迭代后放大至ε·e^Ω(ηT)量级
- 锚定固定点可显著稳定算法,适配强化学习与大模型微调场景
镜像下降(MD)扩展了梯度下降(GD)在非欧几何中的应用,近年来被用于强化学习的KL正则化策略优化和大模型微调。这引发一个关键问题:MD动态对输入的鲁棒性如何?我们聚焦于初始化敏感性,即初始值本身可能是预训练或对齐过的模型。已知二次正则化的MD(包括GD和马哈拉诺比斯几何)在凸光滑目标下具有稳定性。本文揭示关键反例:一旦正则化项非二次,即使在欧氏范数下条件良好的正则化器,MD仍可能对初始化表现出指数级敏感性。我们构造了一个三维凸光滑目标函数与强凸、光滑、条件良好的正则化器,在步长η下经过T步迭代后,初始ε扰动被放大至min{polylog⁻¹(1/ε), ε·e^Ω(ηT)}。对于单纯形上的标准KL-正则化MD,即使线性目标在高维或接近边界时也会导致初始ε扰动指数级放大。最后,我们证明添加向锚点的Bregman正则项可稳定动态,同时保持优化性能;锚点选择至关重要:仅以初始值为锚点只能部分缓解不稳定性,而固定点锚定则更有效稳定算法。
原文摘要 · Abstract (English)
Mirror Descent (MD) extends Gradient Descent (GD) beyond Euclidean geometry and has recently reappeared as a lens for KL-regularized policy optimization in reinforcement learning and LLM post-training. This raises a basic robustness question, crucial to reproducibility and reliability: how sensitive are MD dynamics to their inputs? We focus on initialization, often itself a pretrained or previously aligned model. Quadratic-regularized MD, including GD and Mahalanobis geometries, is well-known to be stable for convex smooth objectives. We show a sharp contrast: once the regularizer is non-quadratic, MD can be exponentially more sensitive to initialization than GD, even with a well-conditioned regularizer in Euclidean norm. We give a three-dimensional construction with a convex, smooth objective and a strongly convex, smooth, well-conditioned regularizer where an initial $\varepsilon$ perturbation is quickly amplified to $\min\{\text{polylog}^{-1}(1/\varepsilon), \varepsilon e^{Ω(ηT)}\}$ after $T$ iterations of MD with step size $η$. For canonical KL-regularized MD on the simplex, we show that even linear objectives can amplify an initial $\varepsilon$ perturbation exponentially fast in high-dimensional or near-boundary regimes. Finally, we show that adding a Bregman regularization term toward an anchor point can stabilize the dynamics while largely preserving the optimization guarantees, and that the choice of anchor is crucial: anchoring at the initialization only partially mitigates the instability, whereas anchoring at a fixed point yields a more stable mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。