NOVAK统一优化器提升深度网络训练稳定性与精度
NOVAK: Unified adaptive optimizer for deep neural networks
- 融合自适应动量、学习率修正等五种机制的统一框架
- 在多个数据集上超越14种现有优化器，最高精度超基准3.2%
- 适合训练无残差连接的深层网络，特别适用于资源受限场景
本文提出NOVAK，一种模块化梯度优化算法，整合了自适应动量估计、修正学习率调度、解耦权重正则化、多种Nesterov动量变体及前瞻同步机制，构建性能导向的统一框架。采用双模式架构，包含面向生产的精简快速路径。通过定制CUDA核函数，在标准随机优化假设下实现关键操作3-5倍加速，同时保持数值稳定。提供了修正自适应学习率的完整数学推导，设计了内存高效的前瞻机制（将复杂度从O(2p)降至O(p + p/k)），并实现互补组件的协同作用。理论分析证明了收敛性，并阐明了方法的稳定性与方差降低特性。在CIFAR-10、CIFAR-100、ImageNet和ImageNette上的广泛实验证明，NOVAK优于14种主流优化器（如Adam、AdamW、RAdam、Lion、Adan）。在ResNet-50、VGG-16和ViT等架构上，始终达到当前最优准确率，尤其在VGG-16/ImageNette上表现突出，展现更强的结构鲁棒性。结果表明，其核心贡献（尤其是修正机制、解耦衰减和混合动量）对无跳跃连接的深层网络可靠训练至关重要，解决了现有自适应优化方法长期存在的局限。
原文摘要 · Abstract (English)
This work introduces NOVAK, a modular gradient-based optimization algorithm that integrates adaptive moment estimation, rectified learning-rate scheduling, decoupled weight regularization, multiple variants of Nesterov momentum, and lookahead synchronization into a unified, performance-oriented framework. NOVAK adopts a dual-mode architecture consisting of a streamlined fast path designed for production. The optimizer employs custom CUDA kernels that deliver substantial speedups (3-5 for critical operations) while preserving numerical stability under standard stochastic-optimization assumptions. We provide fully developed mathematical formulations for rectified adaptive learning rates, a memory-efficient lookahead mechanism that reduces overhead from O(2p) to O(p + p/k), and the synergistic coupling of complementary optimization components. Theoretical analysis establishes convergence guarantees and elucidates the stability and variance-reduction properties of the method. Extensive empirical evaluation on CIFAR-10, CIFAR-100, ImageNet, and ImageNette demonstrates NOVAK superiority over 14 contemporary optimizers, including Adam, AdamW, RAdam, Lion, and Adan. Across architectures such as ResNet-50, VGG-16, and ViT, NOVAK consistently achieves state-of-the-art accuracy, and exceptional robustness, attaining very high accuracy on VGG-16/ImageNette demonstrating superior architectural robustness compared to contemporary optimizers. The results highlight that NOVAKs architectural contributions (particularly rectification, decoupled decay, and hybrid momentum) are crucial for reliable training of deep plain networks lacking skip connections, addressing a long-standing limitation of existing adaptive optimization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。