AdaAct通过自适应调整学习率提升神经元输出稳定性，增强模型泛化能力。
An Adaptive Method Stabilizing Activations for Enhanced Generalization
- 根据神经元激活方差动态调整学习率，实现逐神经元自适应优化
- 在CIFAR和ImageNet上表现媲美最先进方法，兼顾收敛速度与泛化性能
- 适合追求高泛化能力且对训练效率有要求的深度学习应用
我们提出AdaAct，一种新型优化算法，依据激活方差自适应调整学习率。该方法在训练过程中引入神经元级自适应机制，增强神经元输出的稳定性，从而提升模型泛化能力，是对传统激活正则化方法的补充。实验表明，AdaAct在标准图像分类基准测试中表现优异。我们在CIFAR和ImageNet上将其与其他前沿方法进行对比，结果表明其有效弥合了Adam的快速收敛与SGD强泛化能力之间的差距，同时保持了具有竞争力的执行时间。代码已公开于https://github.com/hseung88/adaact。
原文摘要 · Abstract (English)
We introduce AdaAct, a novel optimization algorithm that adjusts learning rates according to activation variance. Our method enhances the stability of neuron outputs by incorporating neuron-wise adaptivity during the training process, which subsequently leads to better generalization -- a complementary approach to conventional activation regularization methods. Experimental results demonstrate AdaAct's competitive performance across standard image classification benchmarks. We evaluate AdaAct on CIFAR and ImageNet, comparing it with other state-of-the-art methods. Importantly, AdaAct effectively bridges the gap between the convergence speed of Adam and the strong generalization capabilities of SGD, all while maintaining competitive execution times. Code is available at https://github.com/hseung88/adaact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。