局部学习能逼近反向传播自监督学习的性能。
Can Local Learning Match Self-Supervised Backpropagation?
- 从线性网络理论出发，设计新算法使局部学习精确匹配全局反向传播更新。
- 在非线性卷积网络中，改进后的局部学习在多个图像数据集上达到与全局方法相当的精度。
- 该研究为无需反向传播的高效训练提供新思路，适合关注神经网络优化机制的研究者。
尽管端到端的自监督学习结合反向传播（全局BP-SSL）已成为现代AI系统训练的核心方法，但局部自监督学习（local-SSL）的理论仍难以在深度神经网络中构建有效表征。本文首先针对深层线性网络建立理论：识别出局部-SSL算法（如Forward-forward或CLAPP）实现与全局BP-SSL完全相同权重更新的条件。基于这些理论洞察，我们提出新的局部-SSL算法变体，以近似深度非线性卷积神经网络中的全局BP-SSL。改进后更接近全局梯度更新的局部-SSL方法，在CIFAR-10、STL-10和Tiny ImageNet数据集上表现更优。其中采用CLAPP损失函数的最佳局部-SSL规则，性能媲美使用InfoNCE或CPC类损失的全局BP-SSL，并在这些基准上超越现有最优局部自监督学习方法。
原文摘要 · Abstract (English)
While end-to-end self-supervised learning with backpropagation (global BP-SSL) has become central for training modern AI systems, theories of local self-supervised learning (local-SSL) have struggled to build functional representations in deep neural networks. To establish a link between global and local rules, we first develop a theory for deep linear networks: we identify conditions for local-SSL algorithms (like Forward-forward or CLAPP) to implement exactly the same weight update as a global BP-SSL. Starting from the theoretical insights, we then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks. Variants that improve the similarity between gradient updates of local-SSL with those of global BP-SSL also show better performance on image datasets (CIFAR-10, STL-10, and Tiny ImageNet). The best local-SSL rule with the CLAPP loss function matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions, and improves upon state-of-the-art for local SSL on these benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。