用脉冲神经网络实现低功耗多模态大模型，兼顾性能与能效。
SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression
- 通过模态特异性时间尺度和脉冲压缩，降低多模态输入的时间步开销。
- 在4个主流模型上实现近无损性能，时间步压缩比达3/4时误差仅0.72%~1.19%。
- 适配神经形态硬件，算法硬件协同设计提升能效25.8倍，适合边缘部署。
多模态大语言模型（MLLMs）虽取得显著进展，但推理时计算开销大、能耗高，限制其在资源受限环境中的部署。脉冲神经网络（SNNs）凭借稀疏事件驱动计算，在神经形态硬件上具有天然的能效优势，但将其扩展至MLLM面临两大挑战：异构模态导致统一脉冲编码不足，高分辨率图像输入加剧时间步展开开销。我们提出SpikeMLLM，首个基于脉冲的多模态大模型框架，将现有ANN量化方法统一到脉冲表示空间，并引入由模态演化差异（MED）指导的模态特异性时间尺度（MSTS）及时间压缩的LIF（TC-LIF），将时间步从T=L-1压缩至T=log₂(L)-1。在四个代表性MLLM及多样化的多模态基准测试中，SpikeMLLM在极端时间步压缩（Tv/Tt=3/4）下保持近无损性能，与FP16基线相比，InternVL2-8B和Qwen2VL-72B平均差距仅为0.72%和1.19%。我们进一步设计专用RTL加速器，针对脉冲数据通路优化，在面向部署的协同设计场景下，相较FP16 GPU基线实现9.06倍更高吞吐与25.8倍更好能效，表明算法-硬件协同设计在高效多模态智能中的巨大潜力。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved remarkable progress but incur substantial computational overhead and energy consumption during inference, limiting deployment in resource-constrained environments. Spiking Neural Networks (SNNs), with their sparse event-driven computation, offer inherent energy efficiency advantages on neuromorphic hardware, yet extending them to MLLMs faces two key challenges: heterogeneous modalities make uniform spike encoding insufficient, and high-resolution image inputs amplify timestep unfolding overhead. We propose SpikeMLLM, the first spike-based framework for MLLMs, which unifies existing ANN quantization methods in the spiking representation space and incorporates Modality-Specific Temporal Scales (MSTS) guided by Modality Evolution Discrepancy (MED) and Temporally Compressed LIF (TC-LIF) for timestep compression from T=L-1 to T=log2(L)-1. Experiments on four representative MLLMs across diverse multimodal benchmarks show that SpikeMLLM maintains near-lossless performance under aggressive timestep compression (Tv/Tt=3/4), with average gaps of only 0.72% and 1.19% relative to the FP16 baseline on InternVL2-8B and Qwen2VL-72B. We further develop a dedicated RTL accelerator tailored to the spike-driven datapath, observing 9.06x higher throughput and 25.8x better power efficiency relative to an FP16 GPU baseline under a deployment-oriented co-design setting, suggesting the promise of algorithm-hardware co-design for efficient multimodal intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。