通过限制扰动在神经流形内,提升神经网络的梯度分配效率与生物合理性。
Credit Assignment via Neural Manifold Noise Correlation
- 将扰动限制在低维神经流形上进行梯度估计
- 在CIFAR-10和ImageNet模型上显著提升训练效率
- 结果更接近灵长类视觉系统,支持生物可实现学习机制
信用分配——即单个神经元或突触的变化如何影响网络输出——是大脑与机器学习的核心问题。噪声相关性通过关联活动扰动与输出变化来估计梯度,是一种生物合理的解决方案,但其性能随网络规模增长而急剧下降,因准确估计雅可比矩阵需与网络规模相当的扰动数。此外,各向同性噪声与神经活动位于低维流形的神经生物学观察相悖。为此,我们提出神经流形噪声相关性(NMNC),仅在神经流形上施加扰动进行信用分配。我们从理论上和实证上证明:在训练后的网络中,雅可比行空间与神经流形对齐,且流形维度随网络规模缓慢增长。在卷积网络(CIFAR-10)、ImageNet级模型及循环网络中,NMNC相比传统噪声相关性显著提升性能与样本效率,并生成更接近灵长类视觉系统的表征。这些发现为生物电路如何实现信用分配提供了机制假设,提示生物启发约束可能促进而非限制大规模有效学习。
原文摘要 · Abstract (English)
Credit assignment--how changes in individual neurons and synapses affect a network's output--is central to learning in brains and machines. Noise correlation, which estimates gradients by correlating perturbations of activity with changes in output, provides a biologically plausible solution to credit assignment but scales poorly as accurately estimating the Jacobian requires that the number of perturbations scale with network size. Moreover, isotropic noise conflicts with neurobiological observations that neural activity lies on a low-dimensional manifold. To address these drawbacks, we propose neural manifold noise correlation (NMNC), which performs credit assignment using perturbations restricted to the neural manifold. We show theoretically and empirically that the Jacobian row space aligns with the neural manifold in trained networks, and that manifold dimensionality scales slowly with network size. NMNC substantially improves performance and sample efficiency over vanilla noise correlation in convolutional networks trained on CIFAR-10, ImageNet-scale models, and recurrent networks. NMNC also yields representations more similar to the primate visual system than vanilla noise correlation. These findings offer a mechanistic hypothesis for how biological circuits could support credit assignment, and suggest that biologically inspired constraints may enable, rather than limit, effective learning at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。