通过保持方向一致性提升模型融合效果,避免知识丢失。
DC-Merge: Improving Model Merging with Directional Consistency
- 平衡各任务向量的奇异值能量分布,防止弱但重要的知识被忽略。
- 将向量投影到共享正交子空间对齐方向,减少几何扭曲。
- 在视觉与多模态任务上表现领先,适合需要融合多个适配模型的场景。
模型融合旨在将多个任务适配模型整合为统一模型,同时保留各任务的知识。本文指出,知识保留的关键在于保持合并后多任务向量与单任务向量之间奇异空间的方向一致性。然而,这一一致性常受两个问题影响:一是任务向量内部能量分布不均,少数奇异值占据主导,导致语义重要但较弱的成分在融合中被忽略;二是参数空间中任务向量的几何不一致,直接融合会扭曲其方向结构。为此,我们提出DC-Merge方法,首先通过平滑奇异值平衡各任务向量的能量分布,确保所有知识成分得到充分表达;随后将这些能量平衡后的向量投影至共享正交子空间,对齐其方向几何并最小化重构误差;最后在共享子空间中聚合向量,并投影回原参数空间。在视觉与视觉-语言基准上的大量实验表明,DC-Merge在全微调和LoRA设置下均实现当前最优性能。代码已开源:https://github.com/Tobeginwith/DC-Merge。
原文摘要 · Abstract (English)
Model merging aims to integrate multiple task-adapted models into a unified model that preserves the knowledge of each task. In this paper, we identify that the key to this knowledge retention lies in maintaining the directional consistency of singular spaces between merged multi-task vector and individual task vectors. However, this consistency is frequently compromised by two issues: i) an imbalanced energy distribution within task vectors, where a small fraction of singular values dominate the total energy, leading to the neglect of semantically important but weaker components upon merging, and ii) the geometric inconsistency of task vectors in parameter space, which causes direct merging to distort their underlying directional geometry. To address these challenges, we propose DC-Merge, a method for directional-consistent model merging. It first balances the energy distribution of each task vector by smoothing its singular values, ensuring all knowledge components are adequately represented. These energy-balanced vectors are then projected onto a shared orthogonal subspace to align their directional geometries with minimal reconstruction error. Finally, the aligned vectors are aggregated in the shared orthogonal subspace and projected back to the original parameter space. Extensive experiments on vision and vision-language benchmarks show that DC-Merge consistently achieves state-of-the-art performance in both full fine-tuning and LoRA settings. The implementation code is available at https://github.com/Tobeginwith/DC-Merge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。