发现并剔除影响合并效果的负向LoRA模块,提升多任务模型性能。
Evolutionary Negative Module Pruning for Better LoRA Merging

- 通过进化搜索定位有害的LoRA模块并剔除。
- 在语言与视觉任务上均超越现有合并方法表现。
- 适合需要高效多任务部署的研究者使用。
将多个低秩适配(LoRA)专家合并为单一主干模型,是实现高效多任务部署的有前景方法。现有方法通过权重插值或子空间对齐缓解干扰,但隐含假设所有LoRA矩阵均对合并模型有正向贡献。本文揭示当前合并范式的关键瓶颈:存在‘负向模块’——特定LoRA层在合并后会降低整体性能。为此提出可即插即用的进化负模块剪枝(ENMP)方法,在合并前定位并排除这些有害模块。通过进化搜索策略,有效处理模块选择的离散非可导空间,找到最优剪枝配置。大量实验表明,ENMP持续提升现有合并算法性能,在语言与视觉领域均达到新最佳水平。代码已开源:https://github.com/CaoAnda/ENMP-LoRAMerging。
原文摘要 · Abstract (English)
Merging multiple Low-Rank Adaptation (LoRA) experts into a single backbone is a promising approach for efficient multi-task deployment. While existing methods strive to alleviate interference via weight interpolation or subspace alignment, they rest upon the implicit assumption that all LoRA matrices contribute constructively to the merged model. In this paper, we uncover a critical bottleneck in current merging paradigms: the existence of $\textit{negative modules}$ -- specific LoRA layers that inherently degrade global performance upon merging. We propose $\textbf{E}$volutionary $\textbf{N}$egative $\textbf{M}$odule $\textbf{P}$runing ($\textbf{ENMP}$), a plug-and-play LoRA pruning method to locate and exclude these detrimental modules prior to merging. By leveraging an evolutionary search strategy, ENMP effectively navigates the discrete, non-differentiable landscape of module selection to identify optimal pruning configurations. Extensive evaluations demonstrate that ENMP consistently boosts the performance of existing merging algorithms, achieving a new state-of-the-art across both language and vision domains. Code is available at https://github.com/CaoAnda/ENMP-LoRAMerging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。