用空间位置优化神经元连接,大幅减少参数量。
Training Neural Networks by Optimizing Neuron Positions
- 神经元在欧氏空间中定位,权重由距离反比决定。
- MNIST上性能媲美传统网络,80%以上剪枝仍保持稳定。
- 适合资源受限场景,结构可视化直观易懂。
深度神经网络的高计算复杂度和不断增加的参数量给边缘设备或实时系统部署带来挑战。为此,我们提出一种参数高效的神经架构,将神经元嵌入欧氏空间。训练时优化其位置,连接权重由神经元间空间距离的倒数决定。这种基于距离的布线规则取代了传统的可学习权值矩阵,显著减少参数量,同时引入生物启发的归纳偏置:连接强度随空间距离增加而减弱,反映了大脑在三维空间中连接趋于最短化的特性。我们在多层感知机和脉冲神经网络上验证该方法。实验表明,这些空间嵌入神经网络在MNIST数据集上性能与传统架构相当。此外,模型在剪枝率超过80%的稀疏条件下仍保持良好性能,优于具有相同参数量的传统网络。最后,空间嵌入框架提供了直观的网络结构可视化。
原文摘要 · Abstract (English)
The high computational complexity and increasing parameter counts of deep neural networks pose significant challenges for deployment in resource-constrained environments, such as edge devices or real-time systems. To address this, we propose a parameter-efficient neural architecture where neurons are embedded in Euclidean space. During training, their positions are optimized and synaptic weights are determined as the inverse of the spatial distance between connected neurons. These distance-dependent wiring rules replace traditional learnable weight matrices and significantly reduce the number of parameters while introducing a biologically inspired inductive bias: connection strength decreases with spatial distance, reflecting the brain's embedding in three-dimensional space where connections tend to minimize wiring length. We validate this approach for both multi-layer perceptrons and spiking neural networks. Through a series of experiments, we demonstrate that these spatially embedded neural networks achieve a performance competitive with conventional architectures on the MNIST dataset. Additionally, the models maintain performance even at pruning rates exceeding 80% sparsity, outperforming traditional networks with the same number of parameters under similar conditions. Finally, the spatial embedding framework offers an intuitive visualization of the network structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。