CORAM通过奇异值分解实现模型合并,提升融合效果与稳定性。
CORAM: Coherent Orthogonal Rotation for Model Merging

- 将权重矩阵分块,基于基础模型SVD框架进行任务特异性因子融合。
- 引入放大系数λ,使合并更新收缩后恢复,提升性能0.25至1.35点。
- 适用于多专家语言与视觉语言模型合并,尤其适合大模型微调场景。
模型合并可在不进行联合训练或访问原始数据的情况下整合微调模型的专长。现有方法多在欧氏权重空间中进行线性运算,无法保留更新的几何结构。正交模型合并(OrthoMerge)使用每个权重矩阵的单一正交变换,但无法改变奇异值。本文提出CORAM,将目标矩阵分割为行切片,以基础模型的SVD基底表示每个专家切片的奇异值分解,并在对应流形上融合任务特异性因子。由于流形平均会压缩合并后的更新,CORAM引入放大系数λ=κπ̂c,其中尺度̂c由专家和合并更新范数估计,约等于√N(N个更新幅度相近的专家)。恢复强度κ从专家更新方差中选取,无需评估候选模型,该规则在所有测试套件中与最优扫查值相差不超过0.72分。CORAM还包含扩散切片策略,将高度更新的行分散到各切片,并设置残差路径处理非目标层。在涵盖三种模型族、3B至9B规模、语言与视觉-语言专家的四个套件中,CORAM相较OrthoMerge提升0.25至1.35分,达到或超越最强的权重空间基线。
原文摘要 · Abstract (English)
Merging finetuned models combines specialized capabilities without joint training or access to the original data. Most methods operate by linear arithmetic in Euclidean weight space, which cannot carry the geometry of the update. Orthogonal Model Merging (OrthoMerge) uses a single orthogonal transform for each weight matrix, but such a transform cannot change singular values. We propose CORAM, which partitions each target matrix into row slices, represents every expert slice by its singular value decomposition in the corresponding base-model SVD frame, and merges the task-specific factors on their corresponding manifolds. Because manifold averaging contracts the merged update, CORAM applies an amplification coefficient $λ=κ\hat{c}$. The scale c_hat is estimated from the expert and merged update norms and is approximately $\sqrt{N}$ for $N$ experts with comparable update magnitudes. The restoration strength kappa is selected from the dispersion of expert updates without evaluating candidate merged models. This rule remains within 0.72 points of the best swept value on all evaluated suites. CORAM also includes spread slicing to distribute highly updated rows across slices and a residual pathway for non-target layers. Across four suites covering three model families, 3B to 9B scales, and language and vision-language experts, CORAM improves over OrthoMerge by 0.25 to 1.35 points and matches or exceeds the strongest weight-space baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。