针对Mamba模型提出高效微调方法，仅动1%参数就达优秀效果
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation
- 聚焦预训练的投影层而非状态空间模型进行微调
- 仅用不到总参数1%实现跨视觉与语言任务的强性能
- 适合资源受限场景下快速适配Mamba模型
尽管Mamba架构作为Transformer的潜在替代者日益受到关注，但针对Mamba的参数高效微调（PEFT）方法仍鲜有研究。本文基于两项关键发现：(1) 尽管状态空间模型（SSMs）被视为Mamba的核心，但在迁移学习中起主导作用的其实是投影层（Projectors）；(2) 基于此，我们提出一种专为Mamba设计的新型PEFT方法——投影层目标型对角中心线性变换（ProDiaL）。该方法仅通过中心对角线性变换矩阵优化预训练投影层，不直接更新投影权重，实现高效任务适配。实验表明，该方法仅使用少于总参数1%的量，在多种视觉与语言类Mamba模型上均表现出色，展现出卓越的通用性与有效性。
原文摘要 · Abstract (English)
Despite the growing interest in Mamba architecture as a potential replacement for Transformer architecture, parameter-efficient fine-tuning (PEFT) approaches for Mamba remain largely unexplored. In our study, we introduce two key insights-driven strategies for PEFT in Mamba architecture: (1) While state-space models (SSMs) have been regarded as the cornerstone of Mamba architecture, then expected to play a primary role in transfer learning, our findings reveal that Projectors -- not SSMs -- are the predominant contributors to transfer learning. (2) Based on our observation, we propose a novel PEFT method specialized to Mamba architecture: Projector-targeted Diagonal-centric Linear Transformation (ProDiaL). ProDiaL focuses on optimizing only the pretrained Projectors for new tasks through diagonal-centric linear transformation matrices, without directly fine-tuning the Projector weights. This targeted approach allows efficient task adaptation, utilizing less than 1% of the total parameters, and exhibits strong performance across both vision and language Mamba models, highlighting its versatility and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。