提出新方法合并LoRA适配器,提升多任务性能。
CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging
- 基于任务子空间投影平均构建共识方向
- 在ViT-B/32上比DC-Merge高2.56分,ViT-L/14高1.51分
- 适合需要高效部署多任务适配器的场景
LoRA适配器可高效定制预训练模型以适应多个下游任务,但为每个任务部署独立适配器需占用存储空间,并在推理时进行任务选择。模型合并通过将独立训练的适配器整合为单一多任务适配器解决此问题。现有基于SVD的合并方法主要关注共享或任务特异性方向的构造,而最终方向的系数通常直接沿用原始任务SVD结果。在固定合并基底下,继承系数虽保持组件顺序与高秩相关性,但其大小与任务更新诱导的系数差异显著。为缓解这一偏差,本文提出CT-Merging,一种面向LoRA的合并算法:从平均任务子空间投影中估计共识方向,并在最终更新中分配任务级均方根系数尺度。该方法利用任务SVD子空间间的重复支持构建公共基底,减少对秩级别SVD幅度的依赖。在DC-Merge CLIP适配器基准测试中,CT-Merging在平均归一化准确率上优于当前最优方法,在ViT-B/32和ViT-L/14 KnoTS训练检查点上分别较DC-Merge提升2.56和1.51点。
原文摘要 · Abstract (English)
LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time. Model merging addresses this issue by combining independently trained adapters into one multi-task adapter. Recent SVD-based LoRA merging methods mainly focus on constructing shared or task specific directions, while the coefficients assigned to the final directions are often directly from the original task SVD. On a fixed merged basis, inherited coefficients preserve component order with high rank correlation, yet their magnitudes differ substantially from the coefficients induced by the task updates. To address this mismatch, we propose CT-Merging, a LoRA-aware merging algorithm that estimates consensus directions from average task subspace projectors and assigns task-level RMS coefficient scales in the final update. CT-Merging uses repeated support across task SVD subspaces to construct the common basis, while reducing reliance on rank wise SVD magnitudes after direction construction. On the DC-Merge CLIP adapter benchmark, CT-Merging achieves superior average normalized accuracy compared to state-of-the-art merging methods and further improves over DC-Merge by 2.56 points on ViT-B/32 and 1.51 points on ViT-L/14 KnoTS-trained checkpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。