针对状态空间模型提出高效微调方法，显著提升性能。
Parameter-Efficient Fine-Tuning of State Space Models
- 设计专用于SSM模块的稀疏维度微调方法
- 结合LoRA与SDT，在多个任务上达顶尖效果
- 揭示现有微调方法在SSM上的局限性
深度状态空间模型（如Mamba）已成为语言建模的强大工具，具备高性能和与序列长度线性可扩展的优点。然而，针对基于SSM模型的参数高效微调（PEFT）方法仍研究不足。本文首先探讨两个基本问题：(i) 现有PEFT方法在SSM模型上的表现如何？(ii) 应针对哪些参数进行优化以获得最佳结果？分析表明，LoRA及其变体始终优于其他PEFT方法。尽管LoRA对线性投影矩阵有效，但在SSM模块上表现不佳，但仍优于其他适用于SSM的方法，说明后者存在局限。这凸显了开发专用SSM微调方法的必要性。为此，我们提出专为SSM模块设计的稀疏维度微调（SDT）。将SDT与用于线性投影矩阵的LoRA结合，我们在广泛实验中实现了最先进的性能。
原文摘要 · Abstract (English)
Deep State Space Models (SSMs), such as Mamba (Gu & Dao, 2024), have become powerful tools for language modeling, offering high performance and linear scalability with sequence length. However, the application of parameter-efficient fine-tuning (PEFT) methods to SSM-based models remains largely underexplored. We start by investigating two fundamental questions on existing PEFT methods: (i) How do they perform on SSM-based models? (ii) Which parameters should they target for optimal results? Our analysis shows that LoRA and its variants consistently outperform all other PEFT methods. While LoRA is effective for linear projection matrices, it fails on SSM modules-yet still outperforms other methods applicable to SSMs, indicating their limitations. This underscores the need for a specialized SSM tuning approach. To address this, we propose Sparse Dimension Tuning (SDT), a PEFT method tailored for SSM modules. Combining SDT for SSMs with LoRA for linear projection matrices, we achieve state-of-the-art performance across extensive experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。