提出串行低秩适配，参数减至1/4仍保持性能
Serial Low-rank Adaptation of Vision Transformer
- 将共享低秩矩阵串行融入注意力机制，减少参数冗余
- 参数量仅为LoRA的1/4，多数任务表现相当
- 适用于资源受限场景，适合高效微调视觉模型
在计算和存储资源受限的现实场景下，以参数高效方式微调大型预训练视觉基础模型至关重要。低秩适配（LoRA）通过将参数空间降为低秩形式，已展现出显著效率。然而，在资源受限条件下进一步降低参数与内存需求仍是重大挑战。本文在主流视觉变换器基础上，提出一种新型LoRA变体——串行低秩适配（Serial LoRA），其将共享低秩矩阵串行地嵌入注意力机制中，有效提取适配参数的内在共性，显著减少冗余。值得注意的是，Serial LoRA仅需LoRA 1/4的参数量，却在多数情况下达到相近性能。我们在多种基于Transformer结构的视觉基础模型上进行了广泛实验，结果验证了该方法的一致优势。
原文摘要 · Abstract (English)
Fine-tuning large pre-trained vision foundation models in a parameter-efficient manner is critical for downstream vision tasks, considering the practical constraints of computational and storage costs. Low-rank adaptation (LoRA) is a well-established technique in this domain, achieving impressive efficiency by reducing the parameter space to a low-rank form. However, developing more advanced low-rank adaptation methods to reduce parameters and memory requirements remains a significant challenge in resource-constrained application scenarios. In this study, we consider on top of the commonly used vision transformer and propose Serial LoRA, a novel LoRA variant that introduces a shared low-rank matrix serially composite with the attention mechanism. Such a design extracts the underlying commonality of parameters in adaptation, significantly reducing redundancy. Notably, Serial LoRA uses only 1/4 parameters of LoRA but achieves comparable performance in most cases. We conduct extensive experiments on a range of vision foundation models with the transformer structure, and the results confirm consistent superiority of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。