提出一种无需反向传播的分层学习方法,实现高效低内存训练。
Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation
- 基于马尔可夫假设将全局目标分解为局部层更新,每层独立优化
- 在MNIST到ImageNet上性能接近反向传播,内存不随深度增加
- 适合追求模块化、可扩展训练的科研与工程应用
反向传播支撑现代深度学习,但依赖全局梯度同步限制了可扩展性并带来高内存开销。完全局部的学习规则更高效,却常难以维持跨层协调。我们提出随机分层学习(SLL),将全局目标分解为协同的层内更新,保持全局表征一致性。该方法受证据下界(ELBO)启发,在网络上采用马尔可夫假设,使网络级目标分解为层级项,每层通过确定性编码器优化局部目标。不可计算的KL散度由基于固定几何保持随机投影得到的辅助类别后验的巴氏距离替代,可选乘性丢弃提供随机正则化。SLL局部优化,全局对齐,从而消除跨层反向传播。在MLP、CNN和视觉变换器上从MNIST到ImageNet的实验表明,该方法超越近期局部方法,性能媲美全局反向传播,且内存使用与深度无关。结果展示了一条实用而严谨的模块化、可扩展局部学习路径,实现纯粹局部计算与全局一致表征的耦合。
原文摘要 · Abstract (English)
Backpropagation underpins modern deep learning, yet its reliance on global gradient synchronization limits scalability and incurs high memory costs. In contrast, fully local learning rules are more efficient but often struggle to maintain the cross-layer coordination needed for coherent global learning. Building on this tension, we introduce Stochastic Layer-wise Learning (SLL), a layer-wise training algorithm that decomposes the global objective into coordinated layer-local updates while preserving global representational coherence. The method is ELBO-inspired under a Markov assumption on the network, where the network-level objective decomposes into layer-wise terms and each layer optimizes a local objective via a deterministic encoder. The intractable KL in ELBO is replaced by a Bhattacharyya surrogate computed on auxiliary categorical posteriors obtained via fixed geometry-preserving random projections, with optional multiplicative dropout providing stochastic regularization. SLL optimizes locally, aligns globally, thereby eliminating cross-layer backpropagation. Experiments on MLPs, CNNs, and Vision Transformers from MNIST to ImageNet show that the approach surpasses recent local methods and matches global BP performance while memory usage invariant with depth. The results demonstrate a practical and principled path to modular and scalable local learning that couples purely local computation with globally coherent representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。