提出无需反向传播的训练方法，可任意深度神经网络
Forward Only Learning for Orthogonal Neural Networks of any Depth
- 基于前向传播设计新算法，满足正交性时等价于反向传播
- 实验表明可训练任意深度网络，性能优于PEPITA
- 适合追求低内存消耗的模型训练场景
反向传播仍是当前训练神经网络的主流算法。随着架构规模指数级增长，其计算开销日益成为负担。近期的PEPITA和前向仅框架虽具潜力，但难以扩展至多个隐藏层，限制了应用。本文首先理论分析这些方法的主要局限，据此设计一种前向仅算法，在线性与正交假设下等价于反向传播。通过放松线性假设，进一步提出FOTON（Forward-Only Training of Orthogonal Networks），显著缩小与反向传播的差距。实验表明，FOTON能训练任意深度网络，无需反向传播，且在卷积网络上表现优异，为更复杂架构的应用开辟路径。代码已开源：https://github.com/p0lcAi/FOTON。
原文摘要 · Abstract (English)
Backpropagation is still the de facto algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this algorithm also becomes a burden. The recent PEPITA and forward-only frameworks have proposed promising alternatives, but they failed to scale up to a handful of hidden layers, yet limiting their use. In this paper, we first analyze theoretically the main limitations of these approaches. It allows us the design of a forward-only algorithm, which is equivalent to backpropagation under the linear and orthogonal assumptions. By relaxing the linear assumption, we then introduce FOTON (Forward-Only Training of Orthogonal Networks) that bridges the gap with the backpropagation algorithm. Experimental results show that it outperforms PEPITA, enabling us to train neural networks of any depth, without the need for a backward pass. Moreover its performance on convolutional networks clearly opens up avenues for its application to more advanced architectures. The code is open-sourced at https://github.com/p0lcAi/FOTON .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。