ZENITH自动调整学习率，提速提效还更准。
ZENITH: Automated Gradient Norm Informed Stochastic Optimization
- 根据梯度范数变化动态调整学习率，无需额外计算开销。
- 在6个图像分类任务中，测试准确率更高且耗时更短。
- 兼容正则化，适合需要良好泛化的视觉模型训练。
训练深度计算机视觉模型通常需要人工干预或手动调优学习率（LR）调度。尽管现有自适应优化器可自动调度学习率，但存在计算与内存开销大、不兼容正则化、学习率选择不佳等问题。本文提出ZENITH（Zero-overhead Evolution using Norm-Informed Training History）优化器，通过分析梯度范数的时间演化来自适应调整学习率。在6种CNN架构和6个基准数据集的图像分类实验中，ZENITH在更短的墙钟时间内实现了更高的测试准确率。在使用R-CNN系列模型的MS COCO数据集上，其在目标检测、关键点检测和实例分割任务中也取得了更优的mAP。此外，其对正则化的兼容性进一步提升了模型泛化能力。
原文摘要 · Abstract (English)
Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule. While existing adaptive optimizers schedule the LR automatically, they suffer from computational and memory overhead, incompatibility with regularization, and suboptimal LR choices. In this work, we introduce the ZENITH (Zero-overhead Evolution using Norm-Informed Training History) optimizer, which adapts the LR using the temporal evolution of the gradient norm. Image classification experiments spanning 6 CNN architectures and 6 benchmarks demonstrate that ZENITH achieves higher test accuracy in lower wall-clock time than baselines. It also yielded superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO using the R-CNN family of models. Furthermore, its compatibility with regularization enables even better generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。