提出无损转换方法LAS，实现全脉冲驱动的大语言模型
LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models
- 设计新型神经元解决激活异常与非线性操作不兼容问题
- 在6个语言模型和2个视觉语言模型上实现零性能损失
- 适合追求能效比的脉冲神经网络研究者与硬件部署团队
脉冲大语言模型（Spiking LLMs）通过事件驱动计算展现出比传统LLMs更低的能耗。为有效构建脉冲LLMs，研究人员利用预训练的ANN参数，采用不同ANN到SNN的转换方法，继承SNN的能效优势。然而，现有方法在处理基于ANN的LLMs中的极端激活异常和不兼容的非线性操作时仍存在困难。为此，我们提出一种无损的ANN-SNN转换方法LAS，用于实现完全脉冲驱动的LLMs。具体而言，LAS引入两种新型神经元以解决激活异常和非线性操作问题，并针对脉冲LLMs定制了等效脉冲的Transformer组件，确保完全脉冲转换且无性能损失。在六个语言模型和两个视觉语言模型上的实验结果表明，LAS实现了无损转换。特别地，在OPT-66B模型上，其在WSC任务上的准确率提升了2%。参数分析与消融实验进一步验证了LAS的有效性。源代码已公开于https://github.com/lc783/LAS。
原文摘要 · Abstract (English)
Spiking Large Language Models (LLMs) have emerged as an energy-efficient alternative to conventional LLMs through their event-driven computation. To effectively obtain spiking LLMs, researchers develop different ANN-to-SNN conversion methods by leveraging pre-trained ANN parameters while inheriting the energy efficiency of SNN. However, existing conversion methods struggle with extreme activation outliers and incompatible nonlinear operations of ANN-based LLMs. To address this, we propose a loss-less ANN-SNN conversion for fully spike-driven LLMs, termed LAS. Specifically, LAS introduces two novel neurons to convert the activation outlier and nonlinear operation of ANN-based LLMs. Moreover, LAS tailors the spike-equivalent Transformer components for spiking LLMs, which can ensure full spiking conversion without any loss of performance. Experimental results on six language models and two vision-language models demonstrate that LAS achieves loss-less conversion. Notably, on OPT-66B, LAS even improves the accuracy of 2\% on the WSC task. In addition, the parameter and ablation studies further verify the effectiveness of LAS. The source code is available at https://github.com/lc783/LAS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。