无梯度优化算法在噪声与不规则目标函数下的收敛性分析
Convergence of a class of gradient-free optimisation schemes when the objective function is noisy, irregular, or both
- 用光滑近似替代真实梯度，构建通用无梯度更新框架
- 在弱正则条件下实现收敛，平滑程度与步长存在权衡
- 适用于黑箱、不可导、含噪声的机器学习优化场景
我们研究一类用于最小化可能非光滑、含噪声的目标函数的迭代算法的收敛性，该目标函数可能代数不可解，其值只能通过黑箱获得。这些算法可统一为广义梯度下降形式，其中使用目标函数的光滑近似梯度。所建立的框架包含模型驱动和模糊化方法两种经典零阶优化方法。在对目标函数正则性要求极低的条件下得到收敛结果，且平滑程度与参数更新步长之间存在权衡。在随机情形下需额外假设。通过一个具有挑战性的机器学习分类实例，展示了算法及收敛结果的实际意义。
原文摘要 · Abstract (English)
We investigate the convergence properties of a class of iterative algorithms designed to minimize a potentially non-smooth and noisy objective function, which may be algebraically intractable and whose values may be obtained as the output of a black box. The algorithms considered can be cast under the umbrella of a generalised gradient descent recursion, where the gradient is that of a smooth approximation of the objective function. The framework we develop includes as special cases model-based and mollification methods, two classical approaches to zero-th order optimisation. The convergence results are obtained under very weak assumptions on the regularity of the objective function and involve a trade-off between the degree of smoothing and size of the steps taken in the parameter updates. As expected, additional assumptions are required in the stochastic case. We illustrate the relevance of these algorithms and our convergence results through a challenging classification example from machine learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。