提出统一参数假设,让非凸优化收敛性分析更贴近实际。
A Novel Unified Parametric Assumption for Nonconvex Optimization
- 引入新参数化假设,统一描述多种非凸函数。
- 理论证明梯度法在确定与随机场景下均收敛。
- 可还原经典函数类,适合研究优化算法的学者。
非凸优化是现代机器学习的核心,但现有理论对收敛性的保证过于悲观。虽然凸优化效率高,但适用范围有限。为弥合这一差距并理解实际中优化算法的成功,我们提出一种新型统一参数化假设。该假设足够通用以涵盖广泛的非凸函数,同时又足够具体,能推导出基于梯度方法的统一收敛定理。通过调整参数,我们的假设可还原多个已有函数类,并识别出可高效优化的函数。我们推导了确定性和随机优化下的收敛定理,并通过实验验证该假设在优化轨迹上可实际成立。
原文摘要 · Abstract (English)
Nonconvex optimization is central to modern machine learning, but the general framework of nonconvex optimization yields weak convergence guarantees that are too pessimistic compared to practice. On the other hand, while convexity enables efficient optimization, it is of limited applicability to many practical problems. To bridge this gap and better understand the practical success of optimization algorithms in nonconvex settings, we introduce a novel unified parametric assumption. Our assumption is general enough to encompass a broad class of nonconvex functions while also being specific enough to enable the derivation of a unified convergence theorem for gradient-based methods. Notably, by tuning the parameters of our assumption, we demonstrate its versatility in recovering several existing function classes as special cases and in identifying functions amenable to efficient optimization. We derive our convergence theorem for both deterministic and stochastic optimization, and conduct experiments to verify that our assumption can hold practically over optimization trajectories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。