用有效维度压缩提升前向-前向学习,无需负样本
Stochastic Forward-Forward Learning through Representational Dimensionality Compression
- 以神经响应有效维度替代激活平方和,捕捉神经元间相关性
- 在噪声输入下最小化维度,跨样本最大化维度,性能媲美非反向传播方法
- 揭示噪声的建设性作用,适合类脑计算场景
前向-前向(Forward-Forward, FF)学习为神经网络训练提供了一种替代反向传播的自下而上方案,依赖于逐层的“优度”函数,并通过精心设计的负样本进行对比学习。现有优度函数通常定义为突触后激活的平方和,忽略了神经元间的相关变异性。本文提出一种新型优度函数——维度压缩,利用波动神经响应的有效维度(ED)来引入二阶统计结构。目标是在单个输入的噪声副本中最小化ED,同时在样本分布上最大化ED,从而在无需负样本的情况下促进结构化表示。实验表明该方法性能可与其它非反向传播方法相媲美。此外,我们发现噪声具有建设性作用,当预测基于输出平方均值(等价于能量项)时,能增强泛化能力并改善推理表现。研究结果有助于构建更符合生物学原理的学习算法,并提示其天然适配于类脑计算系统,其中随机性是计算资源而非干扰因素。代码已公开于https://github.com/ZhichaoZhu/StochasticForwardForward。
原文摘要 · Abstract (English)
The Forward-Forward (FF) learning algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise "goodness" function with well-designed negative samples for contrastive learning. Existing goodness functions are typically defined as the sum of squared postsynaptic activations, neglecting correlated variability between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for noisy copies of individual inputs while maximizing it across the sample distribution, promoting structured representations without the need to prepare negative samples.We demonstrate that this formulation achieves competitive performance compared to other non-BP methods. Moreover, we show that noise plays a constructive role that can enhance generalization and improve inference when predictions are derived from the mean of squared output, which is equivalent to making predictions based on an energy term. Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing, where stochasticity is a computational resource rather than a nuisance. The code is available at https://github.com/ZhichaoZhu/StochasticForwardForward
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。