分析带重尾马尔可夫噪声的随机逼近收敛性,给出误差尾部的紧致上界。
Concentration of General Stochastic Approximation Under Heavy-Tailed Markovian Noise
- 构造新李雅普诺夫函数结合泊松方程生成函数分析迭代误差。
- 噪声有界时误差尾部可达亚高斯或比帕累托轻但比威布尔重的类型。
- 适用于研究带复杂噪声的优化算法稳定性,如强化学习和在线学习。
我们为具有通用步长的随机逼近算法在有限状态马尔可夫噪声与鞅差噪声之和的条件下,建立了迭代点的最大集中界限。当鞅差噪声有界时,误差尾部表现为亚高斯、亚威布尔,或比任意帕累托轻但比任意威布尔重的类型,具体取决于步长序列及随机算子是否几乎必然收缩、非扩张或以正概率扩张。分析依赖于一种新颖的李雅普诺夫函数,该函数涉及泊松方程解的矩生成函数,并引入辅助投影算法。我们通过最坏情况示例证明了定性更紧的界不可能存在。进一步研究平均算子收缩且步长为 $1/k$ 时无界鞅差噪声的情形:若算子几乎必然非扩张,则误差尾部至多是噪声尾部的三倍;若算子以正概率扩张,则误差尾部可能显著更重。这些结果通过一种新颖的黑箱截断方法实现,将无界噪声情形归约为有界噪声情形。
原文摘要 · Abstract (English)
We establish maximal concentration bounds for the iterates generated by stochastic approximation algorithms with general step sizes, where the noise has a finite-state Markovian component plus a Martingale-difference component. When the Martingale-difference noise is bounded, we show that the tail of the error can be sub-Gaussian, sub-Weibull, or something lighter than any Pareto but heavier than any Weibull, depending on the step size sequence and on whether the random operator is almost surely contractive, almost surely non-expansive, or expansive with positive probability. Our analysis relies on a novel Lyapunov function involving the moment-generating function of the solution to a Poisson equation, together with an auxiliary projected algorithm. We complement the upper bounds with worst-case examples showing that qualitatively sharper bounds are impossible. We further study the case of unbounded Martingale-difference noise when the average operator is contractive, and the step sizes are of order $1/k$. In this setting, we show that if the random operator is almost surely non-expansive, then the error tail is at most three times heavier than the noise tail, whereas if the random operator is expansive with positive probability, then the error may have substantially heavier tails. These results are obtained through a novel black-box truncation argument that reduces the unbounded-noise setting to the bounded-noise case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。