通过合并多个LoRA适配器，实现视觉多任务模型，无需重新训练。
Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model
- 将多个视觉任务的LoRA适配器简单合并，构建多任务模型。
- 合并后性能略降但可接受，部分组合优于微调分类头。
- 差异大的数据集训练的LoRA合并效果更好，适合高效部署。
参数高效微调（PEFT）方法在大语言模型和计算机视觉生成模型中广泛应用。尤其可在推理阶段使用多个适配器来改变基座模型行为。本文研究了多个针对计算机视觉任务训练的LoRA适配器是否可合并使用而不损失性能。若可行，则仅通过合并不同LoRA即可创建多任务模型，减少推理时间且无需额外训练。我们在六个不同任务上训练了适配器，并评估其合并后的表现。作为对比，采用冻结主干网络并微调其分类头的模型。结果表明，即使使用简单的合并技术，通过合并多个适配器构建多任务模型是可行的，某些情况下性能略有下降。实验中最多合并了三个适配器。根据任务类型和训练数据相似性，合并效果可能优于头部微调。我们观察到，由差异较大的数据集训练的LoRA在合并后表现更优。
原文摘要 · Abstract (English)
Parameter efficient finetuning (PEFT) methods are widely used in LLMs and generative models in computer vision. Especially one can use multiple of these during inference to change the behavior of the base model. In this paper we investigated whether multiple LoRA adapters trained on computer vision tasks can be merged together and used during inference without loss in performance. By achieving this, multitask models can be created just by merging different LoRAs. Merging these will reduce inference time and it will not require any additional retraining. We have trained adapters on six different tasks and evaluated their performance when they are merged together. For comparison we used a model with a frozen backbone and finetuned its head. Our results show that even with simple merging techniques creating a multitask model by merging adapters is achievable by slightly loosing performance in some cases. In our experiments we merged up to three adapters together. Depending on the task and the similarity of the data adapters were trained on, merges can outperform head finetuning. We have observed that LoRAs trained with dissimilar datasets tend to perform better compared to model trained on similar datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。