用脉冲神经网络实现类脑大模型，兼顾性能与能效。
BrainTransformers: SNN-LLM
- 用脉冲神经网络重构Transformer组件，适配生物可解释性计算。
- 30亿参数模型在多项评测中表现接近传统LLM，如GSM8K达76.3。
- 适合关注类脑计算、低功耗AI的科研与工程人员。
本研究提出BrainTransformers，一种基于脉冲神经网络（SNN）构建的大语言模型。核心贡献包括：设计SNN兼容的Transformer组件，如SNNMatmul、SNNSoftmax和SNNSiLU；实现SiLU激活函数的SNN近似；开发Synapsis模块模拟突触可塑性。所提出的30亿参数模型BrainTransformers-3B-Chat在多个基准测试中表现优异，包括MMLU（63.2）、BBH（54.1）、ARC-C（54.3）和GSM8K（76.3），同时具备潜在的更高能效与生物合理性。模型采用三阶段训练策略，包含针对神经元突触可塑性的专门训练。该研究为自然语言处理中的类脑AI系统与神经形态计算开辟了新路径。未来工作将聚焦硬件优化、专用SNN微调工具开发，以及在节能计算环境中的实际应用探索。
原文摘要 · Abstract (English)
This study introduces BrainTransformers, an innovative Large Language Model (LLM) implemented using Spiking Neural Networks (SNN). Our key contributions include: (1) designing SNN-compatible Transformer components such as SNNMatmul, SNNSoftmax, and SNNSiLU; (2) implementing an SNN approximation of the SiLU activation function; and (3) developing a Synapsis module to simulate synaptic plasticity. Our 3-billion parameter model, BrainTransformers-3B-Chat, demonstrates competitive performance across various benchmarks, including MMLU (63.2), BBH (54.1), ARC-C (54.3), and GSM8K (76.3), while potentially offering improved energy efficiency and biological plausibility. The model employs a three-stage training approach, including SNN-specific neuronal synaptic plasticity training. This research opens new avenues for brain-like AI systems in natural language processing and neuromorphic computing. Future work will focus on hardware optimization, developing specialized SNN fine-tuning tools, and exploring practical applications in energy-efficient computing environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。