通过分层加权融合任务与领域LoRA,提升多任务模型性能。
TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging

- 基于深度感知的探针引导分层加权,区分任务与领域信号。
- 在六个科学问答任务上平均准确率达45.2%,领先基线3.6个百分点。
- 无需训练,适合作为高效模型融合工具,尤其适合多任务部署。
将任务型LoRA适配器与领域型LoRA适配器合并为统一模型,是实用但研究不足的挑战。现有方法将两类适配器视为对称,全层使用统一权重。我们观察到,在Transformer架构中,任务与领域适配器存在一致的深度依赖不对称性:领域主导性随层数加深增强,浅层则保留更强的任务相关信号。受此启发,我们提出训练免的TaDA(Task-Domain LoRA Merging)算法,通过校准探针引导的逐层门控和分量级子空间感知融合来利用这一结构。门控机制使用对适配器权重大小不变的探针信号,为每层和投影类型分配独立权重;融合阶段在合并前剔除冲突的奇异方向。TaDA生成标准秩-r LoRA适配器,推理开销为零。在六项科学问答基准(Llama-2-7B)上,平均准确率达45.2%,比DARE-TIES高出3.6个百分点,且在所有六项任务上表现最佳。在六项图像分类基准(ViT-L/16)上,平均准确率达85.9%,优于最强基线,并在三项任务中领先。
原文摘要 · Abstract (English)
Combining a task LoRA adapter with a domain LoRA adapter into a single unified model is a practical yet largely unexplored challenge. Existing methods treat both adapters as symmetric peers, applying uniform weights across all layers. We argue that task and domain adapters exhibit a consistent depth-dependent asymmetry across transformer architectures. Domain dominance increases with layer depth, while shallower layers retain stronger task-relevant signals. Motivated by this observation, we propose $\textbf{TaDA}$ ($\textbf{Ta}$sk-$\textbf{D}$omain LoR$\textbf{A}$ Merging), a training-free algorithm that exploits this structure through calibrated probe-guided per-layer gating and per-component subspace-aware merging. The gating assigns individual weights per layer and projection type using a probe signal proved invariant to adapter weight magnitude. The merging discards conflicting singular directions before combining the remaining components. $\textbf{TaDA}$ produces a standard rank-$r$ LoRA adapter with zero inference overhead. On six scientific QA benchmarks with Llama-2-7B, TaDA achieves an average accuracy of 0.452, outperforming DARE-TIES by +3.6 percentage points and obtaining the best result on all six benchmarks. On six image classification benchmarks with ViT-L/16, TaDA reaches 85.9\% average accuracy, improving over the strongest merging baseline while leading in three of the six individual benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。