无需反向传播，用正交权重修改实现高效深度神经网络训练
Orthogonal Weight Modification Enhances Learning Scalability and Convergence Efficiency without Gradient Backpropagation
- 基于低秩扰动与正交约束，设计新型非反向传播权重更新机制
- 可在超过10层脉冲神经网络上本地训练，收敛更快性能更优
- 仅需常数级并行时间复杂度，适合实时、持续学习的类脑系统
针对反向传播（BP）带来的巨大计算开销，非反向传播方法成为新兴类脑系统中高效学习的有吸引力替代方案。然而，现有方法在效率和可扩展性方面仍面临挑战。受大脑神经表征与动态机制启发，我们提出一种基于扰动的权重修改方法——低秩簇正交（LOCO）。研究发现，基于扰动的算法天然具有低秩特性。在此条件下，正交约束可限制节点扰动梯度估计的方差，从而提升收敛效率。在多个数据集上的大量实验表明，LOCO能够实现迄今最深的脉冲神经网络（超过10层）的本地训练，具备出色的持续学习能力，收敛效率更高，任务性能优于其他类脑非反向传播算法。值得注意的是，LOCO的权重更新仅需O(1)并行时间复杂度，远低于传统BP方法。这为在类脑系统上实现高性能、实时、终身学习提供了新方向。
原文摘要 · Abstract (English)
Recognizing the substantial computational cost of backpropagation (BP), non-BP methods have emerged as attractive alternatives for efficient learning on emerging neuromorphic systems. However, existing non-BP approaches still face critical challenges in efficiency and scalability. Inspired by neural representations and dynamic mechanisms in the brain, we propose a perturbation-based approach called LOw-rank Cluster Orthogonal (LOCO) weight modification. We find that low-rank is an inherent property of perturbation-based algorithms. Under this condition, the orthogonality constraint limits the variance of the node perturbation (NP) gradient estimates and enhances the convergence efficiency. Through extensive evaluations on multiple datasets, LOCO demonstrates the capability to locally train the deepest spiking neural networks to date (more than 10 layers), while exhibiting strong continual learning ability, improved convergence efficiency, and better task performance compared to other brain-inspired non-BP algorithms. Notably, LOCO requires only O(1) parallel time complexity for weight updates, which is significantly lower than that of BP methods. This offers a promising direction for achieving high-performance, real-time, and lifelong learning on neuromorphic systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。