用状态空间模型增强低秩微调，参数减半性能不降。
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model
- 引入状态空间模型连接低秩矩阵，提升参数利用效率。
- GLUE基准上仅用一半参数即达LoRA同等性能。
- 适合长序列任务，结构更利于信息传递。
微调是适配语言模型至下游任务的关键方法，但随着模型规模增大，更新全部参数变得不切实际。参数高效微调（PEFT）方法如低秩微调（LoRA）通过在预训练权重矩阵中引入额外适配参数来应对这一挑战。然而，LoRA在模型不同插入位置的表现差异显著，暴露出因冗余插入导致的参数效率问题。为此，我们提出SSMLoRA（状态空间模型低秩微调），在LoRA基础上引入状态空间模型（SSM）以连接低秩矩阵。该方法确保在更稀疏插入时仍能保持性能。SSMLoRA不仅将输入映射到低秩空间以增强特征提取，还能利用前一低秩空间的计算结果。实验显示，其在通用语言理解评估（GLUE）基准上表现与LoRA相当，但仅需一半参数。此外，由于结构特性，SSMLoRA在处理长序列任务方面展现出潜力。
原文摘要 · Abstract (English)
Fine-tuning is a key approach for adapting language models to specific downstream tasks, but updating all model parameters becomes impractical as model sizes increase. Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), address this challenge by introducing additional adaptation parameters into pre-trained weight matrices. However, LoRA's performance varies across different insertion points within the model, highlighting potential parameter inefficiency due to unnecessary insertions. To this end, we propose SSMLoRA (State Space Model Low-Rank Adaptation), an extension of LoRA that incorporates a State Space Model (SSM) to interconnect low-rank matrices. SSMLoRA ensures that performance is maintained even with sparser insertions. SSMLoRA allows the model to not only map inputs to a low-rank space for better feature extraction but also leverage the computations from the previous low-rank space. Our method achieves comparable performance to LoRA on the General Language Understanding Evaluation (GLUE) benchmark while using only half the parameters. Additionally, due to its structure, SSMLoRA shows promise in handling tasks with longer input sequences. .You can find our code here:https://github.com/yuhkalhic/SSMLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。