受生物神经元启发,用可训练的树突结构提升模型性能并压缩模型。
Perforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks
- 在神经网络中引入可训练的树突节点,模拟生物神经元的非线性计算能力。
- 在多个任务上提升准确率,实现显著模型压缩且不损失精度。
- 适合追求高性能与轻量化部署的深度学习研究者和工程师。
人工神经网络中的神经元最初设计时对生物神经元的认知有限。本文提出一种基于生物学启发的改进:生物树突不仅是被动传递信号的通道,还能进行复杂的非线性计算。为此,我们引入“穿孔反向传播”机制,在原始网络训练后,逐步添加独立训练的“树突”节点,其目标是捕捉原始神经元的残余误差。训练完成后冻结树突,再以新增误差信号重新训练原神经元,此过程可迭代多次直至性能满意。该方法成功应用于多个现代PyTorch网络,涵盖多种任务领域,均在保持甚至提升准确率的同时实现显著模型压缩。
原文摘要 · Abstract (English)
The neurons of artificial neural networks were originally invented when much less was known about biological neurons than is known today. Our work explores a modification to the core neuron unit to make it more parallel to a biological neuron. The modification is made with the knowledge that biological dendrites are not simply passive activation funnels, but also compute complex non-linear functions as they transmit activation to the cell body. The paper explores a novel system of ``perforated'' backpropagation empowering the artificial neurons of deep neural networks to achieve better performance coding for the same features they coded for in the original architecture. After an initial network training phase, additional ``dendrite'' nodes are added to the network and separately trained with a different objective: to correlate their output with the remaining error of the original neurons. The trained dendrites are then frozen, and the original neurons are further trained, now taking into account the additional error signals provided by the dendrites. The cycle of training the original neurons and then adding and training dendrites can be repeated several times until satisfactory performance is achieved. Our algorithm was successfully added to modern state-of-the-art PyTorch networks across multiple domains, improving upon original accuracies and allowing for significant model compression without a loss in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。