FINDER是一种无需梯度的优化器，结合噪声搜索与牛顿法优势，适合高维非凸优化。
FINDER: Stochastic Mirroring of Noisy Quasi-Newton Search and Deep Network Training
- 基于随机滤波方程设计无梯度更新，模拟牛顿法逆海森矩阵作用
- 在多个基准函数和深度网络任务中，性能优于Adam等主流优化器
- 特别适合高维、非光滑或非凸的工程与深度学习优化问题
本文提出一种针对高维非凸甚至非光滑目标函数的新式随机优化器。该方法旨在融合噪声辅助全局搜索与牛顿类算法的快速局部收敛特性。所提出的方案——名为FINDER（Filtering Informed Newton-like and Derivative-free Evolutionary Recursion）——利用非线性随机滤波方程，推导出一种无梯度更新规则，其形式上类似于使用目标函数逆海森矩阵的牛顿搜索。通过对更新规则进行简化以实现与维度线性扩展，并加入若干改进，将FINDER应用于多种问题：从IEEE基准测试函数，到典型的深度神经网络数据驱动任务，以及部分物理信息深度网络案例。与Adam等知名优化器相比，新方法在多项任务中的表现验证了其在实际高维优化问题中的潜力与前景。
原文摘要 · Abstract (English)
Our proposal is on a new stochastic optimizer for non-convex and possibly non-smooth objective functions typically defined over large dimensional design spaces. Towards this, we have tried to bridge noise-assisted global search and faster local convergence, the latter being the characteristic feature of a Newton-like search. Our specific scheme -- acronymed FINDER (Filtering Informed Newton-like and Derivative-free Evolutionary Recursion), exploits the nonlinear stochastic filtering equations to arrive at a derivative-free update that has resemblance with the Newton search employing the inverse Hessian of the objective function. Following certain simplifications of the update to enable a linear scaling with dimension and a few other enhancements, we apply FINDER to a range of problems, starting with some IEEE benchmark objective functions to a couple of archetypal data-driven problems in deep networks to certain cases of physics-informed deep networks. The performance of the new method vis-á-vis the well-known Adam and a few others bears evidence to its promise and potentialities for large dimensional optimization problems of practical interest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。