揭示脉冲网络节能的极限:任务类型决定能省多少电
The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy
- 通过对比连续与脉冲神经元,发现稀疏性节能受限于任务本质
- 前馈视觉任务可降至5%放电率无损精度,而语言模型最低50%
- 注意力机制虽能大幅压缩,但需以存储开销为代价
脉冲神经网络(SNN)常被视为节能方案,因其稀疏的事件驱动行为替代了密集乘累加。我们提出,这种节能红利并非来自网络本身,而是由任务决定。在保持架构不变的前提下,仅替换隐藏单元类型(连续型与漏积分发放型),并通过双侧目标放电率测试,测量活动水平可降低到何种程度而不影响性能。前馈感知任务可将放电率降至5%而精度无损;而循环语言模型则无法低于约50%,因为递归状态必须持续活跃以传递信息。相比之下,脉冲Transformer可自由压缩至2%(3个种子实验)——说明上限由递归压缩机制而非序列建模决定。注意力机制之所以能突破下限,是因为需存储完整的键值缓存,将放电下限转为内存瓶颈:在类脑硬件上,递归与注意力分别在不同维度付出代价,均无法真正逃脱。我们提出了一个基于信息论的边界公式 ρ ≥ H_b⁻¹(log₂ M / H),并验证其预测:内存负载越高,下限越高;状态宽度越大,下限越低;任务越难,下限也越高(反驳了单纯依赖记忆的观点)。层间输入下限进一步限制了密集输入下的能量压缩空间,从而明确了事件驱动感知是类脑硬件真正的优势领域。
原文摘要 · Abstract (English)
Spiking neural networks (SNNs) are promoted as an energy-efficient substrate because sparse, event-driven activity replaces dense multiply-accumulates with cheap accumulates. We argue the energy dividend of sparsity is not a property of SNNs but of the task. Holding architecture fixed and swapping only the hidden unit (continuous vs. leaky-integrate-and-fire), plus a two-sided target-firing-rate probe, we measure how far activity can be pushed down before quality breaks. Low-load feed-forward perception sparsifies to 5% firing at no accuracy cost; a recurrent language model cannot go below ~50% -- the recurrent state must stay active to carry information. A spiking Transformer, by contrast, sparsifies freely to 2% (3 seeds) -- so the ceiling is a property of recurrent compression, not sequence modeling. Attention escapes the floor only by storing the full key-value cache, trading a firing floor for a memory wall: on neuromorphic hardware, recurrence and attention pay on different axes, neither escapes. We formalize the ceiling with an information-theoretic bound rho >= H_b^{-1}(log2 M / H) and confirm its predictions: the floor rises with memory load, falls with state width, and (refuting a naive memory-only reading) rises with task difficulty. A layer-wise input floor further caps op reduction under dense input, isolating event-driven perception as where neuromorphic hardware wins.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。