为Mamba设计的生物启发微调方法，提升时序建模能力
Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba
- 用类生物膜神经元实现时序信息选择性保留
- 在语言与视觉任务上显著优于现有微调方法
- 适合需要高效适配大模型的下游应用
状态空间模型（SSMs）已成为注意力机制Transformer的强大替代方案，其中Mamba展现出出色的效率与可扩展性。随着模型规模增大，参数高效微调（PEFT）方法对适应下游任务变得至关重要，但避免高昂计算成本。然而，以往方法直接套用针对Transformer设计的PEFT技术，未能考虑SSMs独特的时序处理特性。为此，我们提出专为Mamba设计的膜驱动型微调方法Memba。Memba引入类漏电积分膜（LIM）神经元作为生物启发的门控机制，可自然累积膜电位，增强信息选择性保留能力。通过将LIM神经元与低秩适配（LoRA）及跨层膜电位传递相结合，显著提升了Mamba的时序建模性能。在语言和视觉任务上的大量实验表明，Memba相较于现有方法取得显著改进。代码已公开于https://github.com/Intelligent-Computing-Lab-Yale/Memba。
原文摘要 · Abstract (English)
State Space Models (SSMs) have emerged as powerful alternatives to attention-based Transformers, with Mamba demonstrating impressive efficiency and scalability. As these models grow increasingly larger, the need for Parameter-Efficient Fine-Tuning (PEFT) methods becomes critical to adapt pre-trained Mamba to downstream tasks without prohibitive computational costs. However, previous approaches simply apply traditional Transformer-tailored PEFT methods without addressing the unique temporal processing dynamics of SSMs. To address this limitation, we propose Memba, a membrane-driven PEFT approach specifically designed for Mamba. Memba introduces Leaky Integrate Membrane (LIM) neurons as bio-inspired gating mechanisms that naturally accumulate membrane potentials over time, enhancing selective information retention. By strategically combining LIM neurons with Low-Rank Adaptations (LoRA) and cross-layer membrane transfer, our approach significantly improves Mamba's temporal modeling capabilities. Extensive experiments across language and vision tasks demonstrate that Memba achieves substantial improvements over existing PEFT methods. The code is available at https://github.com/Intelligent-Computing-Lab-Yale/Memba.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。