提出一种利用高阶平滑性的随机优化方法，显著提升收敛速度。
A stochastic first-order method with multi-extrapolated momentum for highly smooth unconstrained optimization
- 每轮迭代进行多次外推，再通过动量更新，利用函数高阶平滑性
- 理论证明样本复杂度达 $\widetilde{\mathcal{O}}(ε^{-(3p+1)/p})$，优于已有结果
- 首次实现任意阶平滑性下的加速，适合高精度优化场景
本文研究具有高阶光滑性的无约束随机优化问题。提出一种新的随机一阶方法（SFOM），在每轮迭代中执行多次外推，随后基于这些外推结果进行动量更新。我们证明该方法可利用目标函数的高阶光滑性实现优化加速。假设目标函数 $f$ 的第 $p$ 阶导数（$p\ge2$）满足Lipschitz连续，在额外温和假设下，该方法在找到满足 $\mathbb{E}[\|\nabla f(x)\|]\leε$ 的点时，达到样本复杂度 $\widetilde{\mathcal{O}}(ε^{-(3p+1)/p})$。据我们所知，这是首个利用任意阶光滑性实现加速的随机一阶方法，其复杂度优于不依赖均方光滑性假设的现有最优结果。初步数值实验验证了方法的实际性能并支持理论结论。
原文摘要 · Abstract (English)
In this paper, we consider an unconstrained stochastic optimization problem where the objective function exhibits high-order smoothness. Specifically, we propose a new stochastic first-order method (SFOM) with multi-extrapolated momentum, in which multiple extrapolations are performed in each iteration, followed by a momentum update based on these extrapolations. We demonstrate that the proposed SFOM can accelerate optimization by exploiting the high-order smoothness of the objective function $f$. Assuming that the $p$th-order derivative of $f$ is Lipschitz continuous for some $p\ge2$, and under additional mild assumptions, we establish that our method achieves a sample complexity of $\widetilde{\mathcal{O}}(ε^{-(3p+1)/p})$ for finding a point $x$ such that $\mathbb{E}[\|\nabla f(x)\|]\leε$. To the best of our knowledge, this is the first SFOM to leverage arbitrary-order smoothness of the objective function for acceleration, resulting in a sample complexity that improves upon the best-known results without assuming the mean-squared smoothness condition. Preliminary numerical experiments validate the practical performance of our method and support our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。