改进脉冲编码方式,让神经网络更省电且性能更强。
Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformers
- 重新分配静默状态,让最常见激活值对应零事件编码
- 在GLUE任务上实现1.64%脉冲率,能耗降低67%
- 适合追求低功耗大模型推理的研究者和工程师
脉冲神经网络(SNNs)有望实现大语言模型的节能推理,但现有节能效果多基于计算操作数,忽略数据移动开销。在22纳米商用芯片上对典型脉冲变压器进行能效分析发现,累加操作能耗不足总能耗的3%,而脉冲触发的核心间传输与权重读取占主导。因此,时间到首次脉冲(TTFS)编码自然成为优选,因其使每个神经元最多仅产生一个脉冲。然而标准TTFS将静默状态(全零脉冲序列)映射为罕见的最小值,而最频繁的激活仍会触发脉冲,违背了节能编码的基本原则——零事件编码应代表最常见值。为此,本文提出掩码时间到首次脉冲编码(M-TTFS),通过时间掩码将静默状态重分配给最常见激活值,并引入死区扩展机制,在可控信息损失下换取更高稀疏性。基于M-TTFS(死区半径k=1)构建的脉冲模型Matterhorn,在GLUE任务上平均得分84.64,整体脉冲率为1.64%,优于此前最佳脉冲模型1.42个百分点,同时能耗降低67%,并在7B至70B参数的脉冲LLaMA模型中持续取得优势。结果表明,在硬件真实计耗下,SNN的能效优势并非必然,而是由与数据分布对齐的编码方式所赢得。
原文摘要 · Abstract (English)
Spiking neural networks (SNNs) promise energy-efficient inference for large language models (LLMs), yet most reported savings rely on compute-operation counts that overlook data movement. Energy characterization of representative spiking transformers on a commercial 22-nm process shows that accumulation contributes less than 3% of total energy, while spike-triggered inter-core transfers and weight reads dominate the cost. This makes time-to-first-spike (TTFS) encoding a natural choice, as it limits each neuron to at most one spike. However, standard TTFS maps the silent state, an all-zero spike train that transmits no events, to the rarely occurring smallest value, while the most common activations still spike. This raises a simple question: why reserve the only cost-free codeword for a rare value? This choice inverts a basic principle of energy-aware coding, under which the zero-event codeword should represent the most common value, rather than a rare extreme. Thus, we introduce masked time-to-first-spike encoding (M-TTFS), which uses a temporal mask to reassign the silent state to the most common activation value, and a dead-zone extension that trades a controlled amount of information for greater sparsity. Built on M-TTFS with dead-zone radius $k{=}1$, our spiking transformer Matterhorn reaches an overall spike rate of 1.64% on GLUE at an average score of 84.64, exceeding the best prior spiking transformer by 1.42 percentage points while consuming 67% less energy, with consistent gains on spiking LLaMA models from 7B to 70B parameters. Together, these results show that under hardware-faithful accounting, the energy advantage of SNNs is not a given: it is earned by encodings that align spikes with the data distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。