提出新方法提升局部学习模型精度,内存仅需反向传播的31%。
Mono-Forward: Revisiting Forward-Forward through Objective-Locality Decomposition
- 将前向-前向算法拆解为局部性与对比目标,分离影响因素
- 新方法在路径MNIST上优于反向传播,内存降低至31%
- 适合追求低内存、高效率训练的深度学习研究者
反向传播仍是深度神经网络训练的主流方法,但存在显著内存开销,且依赖全局误差传播,常被认为缺乏生物合理性。前向-前向(FF)算法是一种有前景的局部学习替代方案,但其精度仍落后于反向传播。一个核心未解问题是:该差距是源于局部性本身,还是源于每层采用的正负样本双通路优度目标?本文通过监督设置下的目标局部性分解重新审视FF。分析表明,性能瓶颈并非仅由局部性导致,也受其优度目标影响。受此启发,我们提出Mono-Forward(MF),简化了FF,保留其局部性,但将对比优度目标替换为标准多分类交叉熵目标,应用于各层,作为在标准分类目标下评估局部学习的控制基线。在MLP和卷积网络上,MF优于原始FF,并保持对多种FF变体的竞争力。在MLP-Mixer上,MF在PathMNIST上的表现强于反向传播,且内存仅需其31%。
原文摘要 · Abstract (English)
Backpropagation remains the dominant algorithm for training deep neural networks, but it incurs substantial memory overhead and relies on global error propagation, which is often regarded as biologically implausible. The Forward-Forward (FF) algorithm is an appealing local-learning alternative to backpropagation, yet it still lags behind backpropagation in accuracy. A central unresolved question is whether this gap arises from FF's locality or from the positive-negative double-pass goodness objective used to train each layer. In this work, we revisit FF under the supervised setting through a decomposition that separates these two design choices. Our analysis suggests that FF's performance limitations are not explained by locality alone, but are also likely influenced by its goodness objective. Motivated by this view, we introduce Mono-Forward (MF), a simplification of FF that preserves its locality while replacing the contrastive goodness objective with a standard multi-class cross-entropy objective applied locally at each layer, serving as a controlled baseline for evaluating local learning under a standard classification objective. Across MLPs and convolutional networks, MF outperforms vanilla FF and remains competitive in multiple FF variants. On MLP-Mixers, MF achieves stronger results on PathMNIST than backpropagation while requiring only 31% of backpropagation's memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。