arXiv:2512.14686cs.LGcs.AI2025-12被引 1

提出适用于重尾噪声的剪裁优化方法,统一了不同噪声强度下的复杂度分析。

Bias-Variance Trade-off for Clipped Stochastic First-Order Methods: From Bounded Variance to Infinite Mean

  • 通过分析梯度剪裁中的偏差-方差权衡,改进了重尾噪声下的优化算法性能。
  • 在尾指数α∈(0,2]范围内,首次获得统一的复杂度界,包含无限均值情形。
  • 理论简洁可扩展,适合研究鲁棒优化与非平稳学习场景的研究者。

随机优化是现代机器学习的基础。近期研究将随机一阶方法(SFOMs)从轻尾噪声拓展到实践中常见的重尾噪声,其中剪裁技术成为控制重尾梯度的关键。已有理论表明,SFOMs的预言复杂度依赖于噪声的尾指数α。然而,现有复杂度结果多局限于α∈(1,2](即噪声均值有限),当α趋近1时复杂度趋于无穷。本文研究了更一般的情形α∈(0,2],涵盖从方差有界到均值无穷的噪声,后者此前研究甚少。通过新提出的梯度剪裁偏差-方差权衡分析,我们证明:当噪声尾部对称性受控时,剪裁后的SFOMs在任意α∈(0,2]下均能获得更优的复杂度保证。该分析不仅给出了覆盖完整尾指数范围的统一复杂度界,且结构清晰,可与轻尾情况的经典分析结合,建立重尾噪声下的预言复杂度结果。数值实验验证了理论发现。

原文摘要 · Abstract (English)

Stochastic optimization is fundamental to modern machine learning. Recent research has extended the study of stochastic first-order methods (SFOMs) from light-tailed to heavy-tailed noise, which frequently arises in practice, with clipping emerging as a key technique for controlling heavy-tailed gradients. Extensive theoretical advances have further shown that the oracle complexity of SFOMs depends on the tail index $α$ of the noise. Nonetheless, existing complexity results often cover only the case $α\in (1,2]$, that is, the regime where the noise has a finite mean, while the complexity bounds tend to infinity as $α$ approaches $1$. This paper tackles the general case of noise with tail index $α\in(0,2]$, covering regimes ranging from noise with bounded variance to noise with an infinite mean, where the latter case has been scarcely studied. Through a novel analysis of the bias-variance trade-off in gradient clipping, we show that when a symmetry measure of the noise tail is controlled, clipped SFOMs achieve improved complexity guarantees in the presence of heavy-tailed noise for any tail index $α\in (0,2]$. Our analysis of the bias-variance trade-off not only yields new unified complexity guarantees for clipped SFOMs across this full range of tail indices, but is also straightforward to apply and can be combined with classical analyses under light-tailed noise to establish oracle complexity guarantees under heavy-tailed noise. Finally, numerical experiments validate our theoretical findings.

随机优化重尾噪声剪裁方法复杂度分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。