用低维误差信号训练大模型,效果媲美传统反向传播。
Training Large Neural Networks With Low-Dimensional Error Feedback
- 用间接低维误差反馈替代高维梯度,解耦前后向传递。
- 任务维度量级的低维误差即可达到反向传播性能。
- 适用于卷积和Transformer,为脑启发学习提供新思路。
深度神经网络训练通常依赖于高维误差信号的反向传播,这一过程计算密集,且缺乏脑科学支持。由于多数任务输出维度较低,我们提出低维误差信号可能足以实现有效学习。为此,我们提出一种基于反馈对齐的新局部学习规则,利用间接、低维误差反馈训练大型网络。该方法将反向传递与前向传递解耦,可在保持高维表征的同时精确控制误差信号维度。我们首先对线性网络进行理论推导,奠定学习框架基础,并将其扩展至非线性、卷积及Transformer架构。令人惊讶的是,仅需与任务维度同阶的极小误差维度,即可达到传统反向传播的性能水平。此外,该规则实现了此前难以应用反馈对齐的卷积网络的高效训练,误差极小。这一突破不仅推动了更生物合理的学习模型发展,也挑战了神经网络训练中对高维梯度信号的依赖。结果表明,低维误差信号可与高维信号同样有效,促使重新审视高维系统中的梯度学习机制。本研究为神经网络优化提供了新视角,并有助于理解人工与生物系统的学习机制。
原文摘要 · Abstract (English)
Training deep neural networks typically relies on backpropagating high dimensional error signals a computationally intensive process with little evidence supporting its implementation in the brain. However, since most tasks involve low-dimensional outputs, we propose that low-dimensional error signals may suffice for effective learning. To test this hypothesis, we introduce a novel local learning rule based on Feedback Alignment that leverages indirect, low-dimensional error feedback to train large networks. Our method decouples the backward pass from the forward pass, enabling precise control over error signal dimensionality while maintaining high-dimensional representations. We begin with a detailed theoretical derivation for linear networks, which forms the foundation of our learning framework, and extend our approach to nonlinear, convolutional, and transformer architectures. Remarkably, we demonstrate that even minimal error dimensionality on the order of the task dimensionality can achieve performance matching that of traditional backpropagation. Furthermore, our rule enables efficient training of convolutional networks, which have previously been resistant to Feedback Alignment methods, with minimal error. This breakthrough not only paves the way toward more biologically accurate models of learning but also challenges the conventional reliance on high-dimensional gradient signals in neural network training. Our findings suggest that low-dimensional error signals can be as effective as high-dimensional ones, prompting a reevaluation of gradient-based learning in high-dimensional systems. Ultimately, our work offers a fresh perspective on neural network optimization and contributes to understanding learning mechanisms in both artificial and biological systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。