无需标签和输出信息,通过压缩空间优化实现多任务LoRA合并
Label-Free Cross-Task LoRA Merging with Null-Space Compression
- 基于适配器几何结构的权重融合,不依赖任务标签或输出
- 在20个异构视觉任务上达到最优性能,均衡提升各任务表现
- 适用于分类、回归与序列生成,适合多任务模型集成场景
模型合并可在不进行联合多任务训练的情况下整合独立微调的检查点。在基础模型时代,低秩适应(LoRA)微调广泛使用,使LoRA合并成为有前景的方向。现有方法在所有任务均为分类的同质设置下有效,但在涉及分类与回归的任务中常失效;基于熵的替代方案不适用于回归,且对大语言模型因长文本序列而成本高昂。本文提出无标签、输出无关的空域压缩(NSC)合并方法,从适配器几何结构中设定合并权重。核心观察发现:在LoRA微调过程中,ΔW = BA中的降维因子A会压缩其零空间,该压缩程度与性能相关。NSC利用此作为优化信号,可泛化至分类、回归及序列生成任务。在二十个异构视觉任务中,NSC取得当前最优表现,各任务收益均衡,优于以往方法对部分任务的过拟合。同时在六个NLI基准、视觉问答与图像描述任务上超越基线,证明其可扩展性与有效性。
原文摘要 · Abstract (English)
Model merging combines independently fine-tuned checkpoints without joint multi-task training. In the era of foundation-model, fine-tuning with Low-Rank Adaptation (LoRA) is prevalent, making LoRA merging a promising target. Existing approaches can work in homogeneous settings where all target tasks are classification but often fail when tasks span classification and regression. Approaches using entropy-based surrogates do not apply to regression and are costly for large language models due to long token sequences. We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry. Our key observation is that during LoRA finetuning the down-projection factor $A$ in $ΔW = BA$ compresses its null space, and the compression correlates with performance. NSC uses this as an optimization signal for merging that can generalize across classification, regression, and sequence generation. NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks. It also outperforms baselines on six NLI benchmarks and on vision-language evaluations for VQA and image captioning, demonstrating scalability and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。