SpikingBrain2.0用脑启发架构实现长文本高效推理，支持跨平台部署。
SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
- 提出双空间稀疏注意力，融合稀疏软注意力与线性注意力。
- 400万上下文下推理速度提升10.13倍，8卡支持超1000万token。
- 适配GPU与神经形态芯片，适合边缘设备和多模态应用。
扩展上下文长度正重塑大模型发展，但全注意力Transformer在长序列下面临计算与推理瓶颈。本文提出SpikingBrain2.0（SpB2.0），一个50亿参数的模型，相比前代在架构与训练效率上均有提升。其核心贡献有二：（1）架构创新：提出双空间稀疏注意力（DSSA），融合稀疏软注意力（MoBA）与稀疏线性注意力（SSE），实现更优的性能-效率权衡；支持两种量化路径：INT8脉冲编码实现稀疏事件驱动计算，FP8编码加速现代GPU推理。（2）训练策略优化：构建针对大语言模型与视觉语言模型的Transformer-to-混合（T2H）转换管道，使用精选开源数据集。实验证明，SpB2.0-5B与SpB2.0-VL-5B仅需不足7000个A100 GPU小时，即可恢复基线Transformer（Qwen3-4B）的大部分能力。在400万上下文下，首次实现10.13倍的首次令牌时间（TTFT）加速，8张A100 GPU可处理超过1000万token，而全注意力模型已超出内存限制。同时具备强跨平台兼容性：在25万上下文下，支持FP8 GPU推理（加速2.52倍），在神经形态硬件上实现64.31%稀疏度，500MHz下面积与功耗分别降低70.6%与46.5%。整体而言，SpikingBrain2.0为轻量级、多模态脉冲基础模型提供实用路径，凸显脑启发机制与高效架构结合在资源受限与边缘场景中的潜力。
原文摘要 · Abstract (English)
Scaling context length is reshaping large-model development, yet full-attention Transformers suffer from prohibitive computation and inference bottlenecks at long sequences. A key challenge is to design foundation models that maintain performance and long-context efficiency with minimal training overhead. We introduce SpikingBrain2.0 (SpB2.0), a 5B model that advances both architecture and training efficiency of its predecessor. Our contributions are two-fold. (1) Architectural Innovation: We propose Dual-Space Sparse Attention (DSSA), an inter-layer hybrid of Sparse Softmax Attention (MoBA) and Sparse Linear Attention (SSE), achieving an improved performance-efficiency trade-off for long-context modeling. SpB2.0 further supports dual quantization paths: INT8-Spiking coding enables sparse event-driven computation, while FP8 coding accelerates inference on modern GPUs. (2) Enhanced Training Strategy: We develop an optimized Transformer-to-Hybrid (T2H) pipeline with dual conversion paths for LLMs and VLMs using curated open-source data. Empirically, SpB2.0-5B and SpB2.0-VL-5B recover most of the base Transformer (Qwen3-4B) capability with under 7k A100 GPU hours. SpB2.0 achieves a 10.13x TTFT speedup at 4M context and supports over 10M tokens on 8 A100 GPUs under vLLM, where full-attention models exceed memory limits. It also demonstrates strong cross-platform compatibility, enabling FP8 GPU inference (2.52x speedup at 250k) and efficient neuromorphic execution (64.31% sparsity, with 70.6% and 46.5% area and power reduction at 500MHz). Overall, SpikingBrain2.0 provides a practical pathway for lightweight, multimodal, spiking foundation models, highlighting the potential of combining brain-inspired mechanisms with efficient architectures for resource-constrained and edge scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。