arXiv:2503.24322cs.LGstat.ML2025-03被引 10

不依赖全网络反向传播,用局部去噪训练神经网络。

NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation

  • 每层独立学习去噪,仅使用本层局部目标和内部反向传播。
  • 在MNIST、CIFAR-10、CIFAR-100上达到可比性能,计算更高效。
  • 适合追求分布式学习或减少梯度传递开销的场景。

传统深度学习通过从输出向参数逐层反向传播误差信号来学习,形成层级表征:顶层为抽象特征,底层为原始特征。本文提出NoProp,一种不依赖全网络前向或反向传播的新方法。受扩散与流匹配启发,每一层独立学习将噪声目标去噪,仅使用本地目标及层内反向传播。该方法需预先固定各层表示为噪声目标版本,学习局部去噪过程,推理时直接应用。在MNIST、CIFAR-10和CIFAR-100图像分类任务上验证有效,结果表明NoProp是可行、易用且计算高效的算法。其脱离全局误差反向传播的范式,改变了网络内信用分配方式,支持更高效的分布式学习,并可能影响学习过程其他特性。

原文摘要 · Abstract (English)

The canonical deep learning approach for learning requires computing a gradient term at each block by back-propagating the error signal from the output towards each learnable parameter. Given the stacked structure of neural networks, where each block builds on the representation of the block below, this approach leads to hierarchical representations. More abstract features live on the top blocks of the model, while features on lower blocks are expected to be less abstract. In contrast to this, we introduce a new learning method named NoProp, which does not rely on either forward or backwards propagation across the entire network. Instead, NoProp takes inspiration from diffusion and flow matching methods, where each block independently learns to denoise a noisy target using only local targets and back-propagation within the block. We believe this work takes a first step towards introducing a new family of learning methods that does not learn hierarchical representations -- at least not in the usual sense. NoProp needs to fix the representation at each block beforehand to a noised version of the target, learning a local denoising process that can then be exploited at inference. We demonstrate the effectiveness of our method on MNIST, CIFAR-10, and CIFAR-100 image classification benchmarks. Our results show that NoProp is a viable learning algorithm, is easy to use and computationally efficient. By departing from the traditional learning paradigm which requires back-propagating a global error signal, NoProp alters how credit assignment is done within the network, enabling more efficient distributed learning as well as potentially impacting other characteristics of the learning process.

神经网络无反向传播去噪学习分布式训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。