解决跨领域模型合并时的子空间冲突问题,提升泛化能力。
Bridging Domains through Subspace-Aware Model Merging
- 通过主成分分析构建共享正交基,缓解不同领域模型间的子空间冲突。
- 在多个架构和规模下,平均性能超越现有合并方法,显著提升跨域泛化能力。
- 适用于需要跨领域迁移的场景,尤其适合多领域微调后的模型整合。
模型合并将多个任务特定模型整合为单一统一模型。尽管近期研究在分布内或多任务场景中提升了合并性能,但模型合并中的领域泛化仍缺乏探索。本文研究在不同领域微调的模型合并对未见领域泛化的影响。通过奇异值分解分析任务矩阵中的参数竞争,发现不同分布偏移下的模型合并会引发比传统多任务设置更强的子空间冲突。为此,提出SCORE(Subspace COnflict-Resolving mErging)方法:通过计算所有模型主导奇异向量的拼接主成分,找到共享正交基,并将每个任务矩阵投影到该基上,剪除非对角成分以消除冲突的奇异方向。SCORE在多种架构与模型规模下均显著优于现有方法,在领域泛化场景中表现出卓越的有效性与可扩展性。
原文摘要 · Abstract (English)
Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task matrix into the shared basis, pruning off-diagonal components to remove conflicting singular directions. SCORE consistently outperforms, on average, existing model merging approaches in domain generalization settings across a variety of architectures and model scales, demonstrating its effectiveness and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。