为稀疏脉冲语言模型设计了高效的CPU推理系统，显著提升速度和内存效率。
Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs
- 将脉冲状态作为执行原语，结合混合内存布局与整数计算优化推理路径。
- 在单线程下达到22.63 tokens/s，内存占用从3.49GB降至1.06GB。
- 适合边缘设备上的低功耗、近传感器推理场景，尤其适用于智能体应用。
脉冲语言模型具有激活稀疏性，而传统Transformer运行时无法直接利用这一特性。本文从系统角度研究该性质，基于SymbolicLight V1脉冲门控语言模型系列，实现了一个C++ CPU推理运行时，将稀疏的二进制脉冲状态作为执行原语，而非仅进行后处理权重压缩。运行时采用显式驱动的权重加载、混合行/列内存布局、AVX2/FMA内核、每通道对称INT8量化及整数域累加，用于脉冲条件下的稀疏路径。在AMD Ryzen 7 5800X上，早期标量FP32基线解码速率为9.5 tokens/s；混合布局的AVX2 FP32提升至14.7 tokens/s，AVX2 INT8在步骤30k导出版本上达19.9 tokens/s，权重体积由3.49 GB减至1.06 GB。对于可用的186k步、874M参数的INT8导出版本，单线程解码速率达22.63 tokens/s，优于TinyLlama-1.1B Q8_0（16.31 tokens/s）、Falcon3-1B Q8_0（11.26 tokens/s）和Qwen2.5-1.5B Q8_0（9.70 tokens/s）。四线程下吞吐达47.90 tokens/s，8线程预填充从29.86提升至94.68 tokens/s。但模型质量有代价：该SNN在WikiText-2上的困惑度为24.80，低于同等基准的密集模型。本文将其视为稀疏语言推理系统的探索，长期目标是服务于具身和边缘智能体，在靠近传感器与执行器处实现本地、低核心推理。脉冲感知执行可改善稀疏脉冲语言模型的CPU吞吐与内存行为，但模型质量、控制性训练基线、具身任务评估与实际能耗测量仍待解决。
原文摘要 · Abstract (English)
Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit. This paper studies that property from a systems perspective. Building on the SymbolicLight V1 spike-gated language model family, we implement a C++ CPU inference runtime that treats sparse binary spike states as an execution primitive rather than only applying post-hoc weight compression. The runtime combines a manifest-driven weight loader, mixed row/column memory layout, AVX2/FMA kernels, per-channel symmetric INT8 quantization, and integer-domain accumulation for spike-conditioned sparse paths. On an AMD Ryzen 7 5800X, an early scalar FP32 baseline decodes at 9.5 tokens/s. Mixed-layout AVX2 FP32 raises this to 14.7 tokens/s, and AVX2 INT8 reaches 19.9 tokens/s on the same step-30k export while reducing the weight footprint from 3.49 GB to 1.06 GB. For the available 186k-step 874M-parameter INT8 export, the C++ runtime decodes at 22.63 tokens/s in a single-thread CPU benchmark, compared with 16.31 tokens/s for TinyLlama-1.1B Q8_0, 11.26 tokens/s for Falcon3-1B Q8_0, and 9.70 tokens/s for Qwen2.5-1.5B Q8_0 under llama.cpp. Thread scaling reaches 47.90 tokens/s at four CPU threads, and 512-token prefill improves from 29.86 to 94.68 tokens/s from one to eight threads. The throughput result comes with a quality cost: the SNN reports WikiText-2 perplexity 24.80, worse than the dense baselines in the same benchmark. We frame the result as an inference-systems study for sparse language runtimes, with longer-term motivation in embodied and edge agents that may benefit from local, low-core inference near sensors and actuators. Spike-aware execution can improve CPU throughput and memory behavior for sparse spiking language models, while model quality, controlled dense training baselines, embodied-task evaluation, and measured CPU energy remain open problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。