通过函数空间优化提升神经元，实现快速收敛的在线学习算法。
Simple Linear Neuron Boosting
- 在函数空间而非参数空间优化神经元，使用预条件梯度下降。
- 算法在多种网络上实现更快的训练速度和更少的训练轮次。
- 无需显式矩阵计算，适合分布式训练，易于实现。
给定可微网络架构和损失函数，本文重新审视基于提升反向传播（Boosted Backpropagation）在函数空间中优化神经元的方法，而非传统参数空间优化。从该视角出发，我们将线性函数空间中的下降过程简化为一种预条件梯度下降算法。我们证明该预条件更新规则等价于对每个神经元特征进行白化重参数化，且归一化操作发生在推理之外。实践中，利用此等价关系，我们构建了一个在线估计器以近似预条件矩阵，并提出一种无需矩阵存储、具有自适应步长的在线学习算法。该方法适用于所有支持自动微分的场景，包括卷积网络和Transformer，且在局部与分布式训练中均易实现。实验表明，该算法在多种任务与网络结构中均实现了快速收敛，无论在训练轮次还是实际耗时方面均表现优异。
原文摘要 · Abstract (English)
Given a differentiable network architecture and loss function, we revisit optimizing the network's neurons in function space using Boosted Backpropagation (Grubb & Bagnell, 2010), in contrast to optimizing in parameter space. From this perspective, we reduce descent in the space of linear functions that optimizes the network's backpropagated-errors to a preconditioned gradient descent algorithm. We show that this preconditioned update rule is equivalent to reparameterizing the network to whiten each neuron's features, with the benefit that the normalization occurs outside of inference. In practice, we use this equivalence to construct an online estimator for approximating the preconditioner and we propose an online, matrix-free learning algorithm with adaptive step sizes. The algorithm is applicable whenever autodifferentiation is available, including convolutional networks and transformers, and it is simple to implement for both the local and distributed training settings. We demonstrate fast convergence both in terms of epochs and wall clock time on a variety of tasks and networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。