模仿大脑神经元群间通信,设计更高效稀疏的神经网络。
Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNs
- 借鉴小鼠视觉皮层的神经群连接模式,构建稀疏模块化网络结构。
- 在多个数据集上实现最高75%稀疏度,准确率反而提升4.3%。
- 适合追求高效率与低计算量的深度学习应用开发者。
生物神经回路具有模块化、分层化和稀疏连接的结构特征,体现了布线成本、功能特化与鲁棒性之间的高效权衡。这些原则为日益加深和扩展的人工神经网络(ANN)设计提供了重要启示。稀疏性已被广泛研究用于降低内存占用、减少计算量、提升速度并增强泛化能力。受系统神经科学发现的启发,本文探索小鼠视觉皮层中神经群间的功能性连接模式如何指导ANN设计。我们提出G2GNet,一种在前馈层中施加稀疏模块化连接的新架构。尽管参数远少于全连接模型,G2GNet在标准视觉基准测试中仍达到更优准确率。据我们所知,这是首个将生物观测到的功能连接模式作为结构偏置引入ANN设计的架构。我们还结合动态稀疏训练(DST)机制,在训练过程中剪枝并重连边,并提出基于激活相关性的赫布式重连规则,借鉴生物可塑性原理。G2GNet在Fashion-MNIST、CIFAR-10和CIFAR-100等基准上实现了高达75%的稀疏度,准确率提升达4.3%,同时计算量显著低于密集基线。
原文摘要 · Abstract (English)
The structure of biological neural circuits-modular, hierarchical, and sparsely interconnected-reflects an efficient trade-off between wiring cost, functional specialization, and robustness. These principles offer valuable insights for artificial neural network (ANN) design, especially as networks grow in depth and scale. Sparsity, in particular, has been widely explored for reducing memory and computation, improving speed, and enhancing generalization. Motivated by systems neuroscience findings, we explore how patterns of functional connectivity in the mouse visual cortex-specifically, ensemble-to-ensemble communication, can inform ANN design. We introduce G2GNet, a novel architecture that imposes sparse, modular connectivity across feedforward layers. Despite having significantly fewer parameters than fully connected models, G2GNet achieves superior accuracy on standard vision benchmarks. To our knowledge, this is the first architecture to incorporate biologically observed functional connectivity patterns as a structural bias in ANN design. We complement this static bias with a dynamic sparse training (DST) mechanism that prunes and regrows edges during training. We also propose a Hebbian-inspired rewiring rule based on activation correlations, drawing on principles of biological plasticity. G2GNet achieves up to 75% sparsity while improving accuracy by up to 4.3% on benchmarks, including Fashion-MNIST, CIFAR-10, and CIFAR-100, outperforming dense baselines with far fewer computations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。