受生物神经元结构启发,提升长序列建模的效率与记忆能力
Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modeling
- 模仿生物神经元树突结构,分枝编码不同频段信号
- 自适应阈值机制降低冗余脉冲,保持训练速度
- 适合边缘设备上的高效长序列处理任务
序列长度的爆炸式增长加剧了对高效长序列建模的需求。得益于内在振荡膜电位动态,共振-放电(Resonate-and-Fire, RF)神经元可高效提取输入信号的频率成分,并将其编码为时空脉冲序列,适用于长序列建模。然而,RF神经元存在有效记忆容量有限、在复杂时序任务中能量效率与训练速度之间存在权衡的问题。受生物神经元树突结构启发,我们提出一种分枝共振-放电(Dendritic Resonate-and-Fire, D-RF)模型,显式引入多树突与胞体架构。每个树突分支利用RF神经元的内在振荡特性编码特定频段,从而实现全面的频率表示。此外,我们在胞体结构中引入自适应阈值机制,根据历史脉冲活动调整阈值,减少冗余脉冲,同时保持长序列任务中的训练效率。大量实验表明,该方法在保持竞争性精度的同时,显著实现稀疏脉冲,且不牺牲训练期间的计算效率。结果凸显其作为边缘平台长序列建模的有效高效解决方案的潜力。
原文摘要 · Abstract (English)
The explosive growth in sequence length has intensified the demand for effective and efficient long sequence modeling. Benefiting from intrinsic oscillatory membrane dynamics, Resonate-and-Fire (RF) neurons can efficiently extract frequency components from input signals and encode them into spatiotemporal spike trains, making them well-suited for long sequence modeling. However, RF neurons exhibit limited effective memory capacity and a trade-off between energy efficiency and training speed on complex temporal tasks. Inspired by the dendritic structure of biological neurons, we propose a Dendritic Resonate-and-Fire (D-RF) model, which explicitly incorporates a multi-dendritic and soma architecture. Each dendritic branch encodes specific frequency bands by utilizing the intrinsic oscillatory dynamics of RF neurons, thereby collectively achieving comprehensive frequency representation. Furthermore, we introduce an adaptive threshold mechanism into the soma structure that adjusts the threshold based on historical spiking activity, reducing redundant spikes while maintaining training efficiency in long sequence tasks. Extensive experiments demonstrate that our method maintains competitive accuracy while substantially ensuring sparse spikes without compromising computational efficiency during training. These results underscore its potential as an effective and efficient solution for long sequence modeling on edge platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。