提出状态偏移微调，让状态空间模型更高效适配下游任务。
State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models
- 直接调整状态变量，而非依赖外部提示
- 在多个数据集上显著优于传统提示微调方法
- 适合追求高效微调的状态空间模型研究者
状态空间模型（SSMs）作为Transformer的高效替代，缓解了其二次计算开销问题。然而，针对SSMs的参数高效微调（PEFT）方法仍鲜有研究。特别是，在Transformer中广泛应用的提示类方法（如Prompt Tuning、Prefix-Tuning）在SSMs上表现不佳。为此，我们提出基于状态的方法作为更优替代。该方法源于SSM的架构特性，直接调整与状态相关的特征，而非依赖外部提示。进一步地，我们提出一种新型基于状态的PEFT方法：状态偏移微调（State-offset Tuning）。该方法在每个时间步直接作用于当前状态，实现更有效的适应。在多个不同数据集上的广泛实验验证了该方法的有效性。代码已开源：https://github.com/furiosa-ai/ssm-state-tuning。
原文摘要 · Abstract (English)
State Space Models (SSMs) have emerged as efficient alternatives to Transformers, mitigating their quadratic computational cost. However, the application of Parameter-Efficient Fine-Tuning (PEFT) methods to SSMs remains largely unexplored. In particular, prompt-based methods like Prompt Tuning and Prefix-Tuning, which are widely used in Transformers, do not perform well on SSMs. To address this, we propose state-based methods as a superior alternative to prompt-based methods. This new family of methods naturally stems from the architectural characteristics of SSMs. State-based methods adjust state-related features directly instead of depending on external prompts. Furthermore, we introduce a novel state-based PEFT method: State-offset Tuning. At every timestep, our method directly affects the state at the current step, leading to more effective adaptation. Through extensive experiments across diverse datasets, we demonstrate the effectiveness of our method. Code is available at https://github.com/furiosa-ai/ssm-state-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。