用脉冲神经网络实现高稀疏激活下的高效语言建模。
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Activation Sparsity
- 通过脉冲-连续双路径结构融合稀疏脉冲与连续残差流。
- 在30亿词中英语料上训练，激活稀疏度超89%，困惑度8.90。
- 适合低功耗部署，生成重复更少，可应用于边缘计算场景。
原生训练的脉冲语言模型需在时间维度保持信息传递的同时使用稀疏二值激活，这导致其性能长期落后于密集型Transformer。本文提出SymbolicLight V1，一种脉冲门控双路径语言模型，结合二值漏电积分-放电（LIF）动态与连续残差流。其双路径稀疏TCAM混合器在连续残差流上采用一阶指数衰减状态与窗口局部注意力，随后是上下文条件解码头。我们在一个包含30亿词、涵盖10个领域的中英文语料上从零训练了四个194M参数模型。在加权保留集上，平均困惑度为8.904（标准差0.019），激活稀疏度超过89%；代码词占比43.7%；十领域未加权平均困惑度为29.38。在相同语料、分词器、令牌预算和硬件条件下，其加权平均困惑度比GPT-2 201M高出7.7%（PPL 8.27）。在五个零样本基准测试中，两个200M规模模型表现无明显差异。在温度0.7、top-k=50采样下，SymbolicLight产生的4元组重复更低；熵调制规则可反转此排名。在实测RTX 2080 Ti上，SymbolicLight每令牌能耗为2,848毫焦，高于GPT-2 201M的905毫焦。
原文摘要 · Abstract (English)
Natively trained spiking language models must preserve information across time while operating through sparse binary activations, a combination that has produced a persistent quality gap relative to dense Transformers. We present SymbolicLight V1, a spike-gated dual-path language model that couples binary Leaky Integrate-and-Fire (LIF) dynamics with a continuous residual stream. Its Dual-Path SparseTCAM mixer combines a first-order exponential-decay state with windowed local attention on the continuous residual stream, followed by a context-conditioned decoding head. We train four 194M-parameter models from scratch on a 3B-token, 10-domain Chinese-English corpus. On a token-weighted held-out set the runs reach PPL 8.88-8.93 (mean 8.904, sample standard deviation 0.019) at more than 89% per-element activation sparsity. Code tokens are 43.7% of that set; the unweighted mean of the ten domain PPLs is 29.38. Under the same corpus, tokenizer, token budget, and hardware, the token-weighted mean is 7.7% above GPT-2 201M (PPL 8.27). Across five zero-shot benchmarks the two 200M-scale models show no clear accuracy separation. Under sampling with temperature 0.7 and top-k 50, SymbolicLight produces lower 4-gram repetition; an entropy-modulated rule reverses that ranking. On a measured RTX 2080 Ti, SymbolicLight uses 2,848 mJ/token versus 905 mJ/token for GPT-2 201M.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。