提出自对比前向算法,让神经网络在资源受限设备上高效自学习。
Self-Contrastive Forward-Forward Algorithm
- 用自生成正负样本替代传统负样本,提升无监督学习稳定性
- 在MNIST、CIFAR-10等数据集上超越现有前向算法性能
- 首次实现前向算法对循环神经网络的训练,适用于序列数据
自主代理系统需具备终身学习能力,但受限于分布式架构下的参数与算力约束,传统训练方法难以适用。前向-前向(Forward-Forward, FF)算法仅依赖前向传播优化各层目标,无需反向传播中的转置操作,具备高效性优势。然而,现有方法在标准基准任务中表现不佳,主要因无监督学习中负样本生成不可靠。本文提出自对比前向算法(Self-Contrastive Forward-Forward, SCFF),受视觉自监督对比学习启发,通过自生成正负样本提升学习效果。SCFF在多个基准数据集(包括MNIST、CIFAR-10、STL-10和Tiny ImageNet)上表现出优于现有无监督局部学习算法的性能。此外,该方法成功扩展至循环神经网络训练,使前向算法可处理序列数据任务。这些成果为资源受限边缘设备上的高精度实时学习提供了可行路径。
原文摘要 · Abstract (English)
Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems, which imposes constraints on both the parameter counts and the computational resources. The Forward-Forward (FF) algorithm is one of these. FF relies only on feedforward operations, the same used for inference, for optimizing layer-wise objectives. This purely forward approach eliminates the need for transpose operations required in traditional backpropagation. Despite its potential, FF has failed to reach state-of-the-art performance on most standard benchmark tasks, in part due to unreliable negative data generation methods for unsupervised learning. In this work, we propose the Self-Contrastive Forward-Forward (SCFF) algorithm, a competitive training method aimed at closing this performance gap. Inspired by standard self-supervised contrastive learning for vision tasks, SCFF generates positive and negative inputs applicable across various datasets. The method demonstrates superior performance compared to existing unsupervised local learning algorithms on several benchmark datasets, including MNIST, CIFAR-10, STL-10, and Tiny ImageNet. We extend FF's application to training recurrent neural networks, expanding its utility to sequential data tasks. These findings pave the way for high-accuracy, real-time learning on resource-constrained edge devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。