自适应立方正则化优化器，无需调参即可高效训练深度模型。
AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning
- 通过动态调整立方项权重，改进牛顿法的收敛稳定性。
- 在图像、语言和信号任务中表现优于主流优化器，且不依赖调参。
- 首次将立方正则化应用于大规模深度学习，适合资源受限场景。
本文提出一种新型正则化技术AdaCubic，可自适应调整立方项权重。其核心是一个带立方约束的辅助优化问题，能动态调节牛顿法中立方项的权重。采用Hutchinson方法近似海森矩阵，显著降低计算开销。实验表明，AdaCubic继承了立方正则化牛顿法的局部收敛性保证。在计算机视觉、自然语言处理和信号处理任务中，其性能优于或媲美多个广泛使用的优化器。与需超参数调优的其他自适应算法不同，AdaCubic仅用一组固定超参数即取得优异效果，适用于无法进行调参的场景。据我们所知，AdaCubic是首个将立方正则化成功应用于可扩展深度学习的优化器。
原文摘要 · Abstract (English)
A novel regularization technique, AdaCubic, is proposed that adapts the weight of the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic constraints that dynamically adjusts the weight of the cubic term in Newton's cubic regularized method. We use Hutchinson's method to approximate the Hessian matrix, thereby reducing computational cost. We demonstrate that AdaCubic inherits the cubically regularized Newton method's local convergence guarantees. Our experiments in Computer Vision, Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic outperforms or competes with several widely used optimizers. Unlike other adaptive algorithms that require hyperparameter fine-tuning, AdaCubic is evaluated with a fixed set of hyperparameters, rendering it a highly attractive optimizer in settings where fine-tuning is infeasible. This makes AdaCubic an attractive option for researchers and practitioners alike. To our knowledge, AdaCubic is the first optimizer to leverage cubic regularization in scalable deep learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。