用训练初期信号预测LoRA适配器能否合并,提前避免干扰。
Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

- 通过低秩更新与梯度对齐度判断合并可行性
- 仅需前几轮训练即预测合并后保留率,准确率达基准最优
- 适合需要高效合并多个任务适配器的部署场景
低秩适配(LoRA)使训练领域和任务特异性语言模型适配器变得廉价,但两个适配器是否可合并通常只有在两者完全训练并评估后才能发现。这种延迟反馈代价高昂:单独表现良好的适配器在合并后可能产生破坏性干扰。我们提出能否提前预判这一结果。将适配器合并能力定义为合并后保持单任务性能的程度,并证明可通过训练初期(前几百分比)测量的信号进行预测——主要考察低秩更新及其梯度在不同任务间的对齐程度,以及对共享表示的扰动程度。我们构建了轻量级预测器MergeProbe,整合这些信号以估计成对或集合级别的性能保留率,并据此作出合并、重加权、剪枝或路由的决策。在涵盖数学、代码、科学、指令遵循和安全性的五领域基准MERGE-PEFT上,MergeProbe在平均和最差情况下的保留率均优于强干扰感知基线,且部署开销远低于完整任务路由。这将LoRA合并从事后工程步骤转变为可预见的测量问题。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated. This late feedback is costly: adapters that are strong in isolation can interfere destructively once their updates are combined. We ask whether this outcome can be anticipated. We formalize adapter mergeability as the degree to which an adapter preserves its single-task utility after merging, and show that it can be forecast from signals measured in the first few percent of training -- chiefly how the low-rank updates and their gradients align across tasks and how much they disturb shared representations. We package these signals into MergeProbe, a lightweight predictor that estimates pairwise and set-level retention and turns the estimate into a concrete decision: merge directly, reweight, prune, or route. On MERGE-PEFT, a five-domain benchmark spanning math, code, science, instruction following, and safety, MergeProbe attains the best average and worst-case retention among strong interference-aware merge baselines while adding far less deployment overhead than full task routing. This turns LoRA merging from a post-hoc engineering step into an anticipatory measurement problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。