arXiv:2410.24216cs.LG2024-10被引 2

让优化器懂网络结构,提速收敛并提升精度

CaAdam: Improving Adam optimizer using connection aware methods

  • 基于层深度、连接数等结构信息动态调整学习率
  • 在CIFAR-10和Fashion MNIST上比Adam更快收敛且更准
  • 无需修改框架,适配现有深度学习系统

我们提出一种受Adam启发的新方法,可提升收敛速度并获得更优的损失最小值。传统优化器(如Adam)在神经网络中采用统一或全局调整的学习率,未考虑网络架构特性。这一架构无关的方法深植于多数深度学习框架中,优化器作为独立模块运行,无法获取网络结构信息。例如,在Keras或PyTorch中,优化器仅依赖梯度和参数,不了解层间连接或拓扑结构。我们的算法CaAdam通过设计结构信息的代理变量,探索这一被忽视的领域,实现连接感知优化。提出多种缩放策略,依据易获取的结构属性(如层深度、连接数量、梯度分布)动态调整学习率。该方法在不改变现有框架的前提下实现更精细的优化。在标准数据集(如CIFAR-10、Fashion MNIST)上的实证评估表明,本方法持续优于标准Adam,收敛更快、准确率更高,证明了在优化策略中引入架构感知的潜力。

原文摘要 · Abstract (English)

We introduce a new method inspired by Adam that enhances convergence speed and achieves better loss function minima. Traditional optimizers, including Adam, apply uniform or globally adjusted learning rates across neural networks without considering their architectural specifics. This architecture-agnostic approach is deeply embedded in most deep learning frameworks, where optimizers are implemented as standalone modules without direct access to the network's structural information. For instance, in popular frameworks like Keras or PyTorch, optimizers operate solely on gradients and parameters, without knowledge of layer connectivity or network topology. Our algorithm, CaAdam, explores this overlooked area by introducing connection-aware optimization through carefully designed proxies of architectural information. We propose multiple scaling methodologies that dynamically adjust learning rates based on easily accessible structural properties such as layer depth, connection counts, and gradient distributions. This approach enables more granular optimization while working within the constraints of current deep learning frameworks. Empirical evaluations on standard datasets (e.g., CIFAR-10, Fashion MNIST) show that our method consistently achieves faster convergence and higher accuracy compared to standard Adam optimizer, demonstrating the potential benefits of incorporating architectural awareness in optimization strategies.

优化器深度学习自适应学习率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。