新方法在重尾噪声下优化更稳定，无需预先知道问题参数。
Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise
- 自适应调整参数，不依赖梯度光滑性等先验信息。
- 在弱平均光滑条件下，找到近似驻点的复杂度最优或最优之一。
- 适合噪声大、非平稳的机器学习场景，如鲁棒训练与强化学习。
本文提出具有Polyak动量、多外推动量和递归动量的实用归一化随机一阶方法，用于求解无约束优化问题。这些方法采用动态更新的算法参数，无需事先知晓问题相关的量（如Lipschitz常数或噪声界）。在重尾噪声和弱平均光滑条件下，我们建立了寻找近似随机驻点的一阶预言机复杂度结果，这两项假设均弱于常用的有界方差和均方光滑性假设。所得复杂度界优于或匹配文献中最优结果。数值实验验证了所提方法的实际有效性。
原文摘要 · Abstract (English)
In this paper, we propose practical normalized stochastic first-order methods with Polyak momentum, multi-extrapolated momentum, and recursive momentum for solving unconstrained optimization problems. These methods employ dynamically updated algorithmic parameters and do not require explicit knowledge of problem-dependent quantities such as the Lipschitz constant or noise bound. We establish first-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise and weakly average smoothness conditions -- both of which are weaker than the commonly used bounded variance and mean-squared smoothness assumptions. Our complexity bounds either improve upon or match the best-known results in the literature. Numerical experiments are presented to demonstrate the practical effectiveness of the proposed methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。