arXiv:2605.26693cs.LGcs.AI2026-05

从几何视角优化模型融合,提升多任务性能

Model Merging on Loss Landscape: A Geometry Perspective

  • 将模型融合建模为流形上的弗雷歇均值,限制在任务向量低秩子空间计算
  • 在八项图像分类任务上,平均准确率与最差任务准确率全面超越基线
  • 统一了曲率感知与谱方法,揭示局部曲率与认知不确定性关联

模型融合为知识整合与并行开发提供了前景广阔的新路径,无需重新训练。然而现有方法或忽略损失曲面的几何特性,或依赖难以计算的全空间海塞近似。本文提出EpiMer框架,将模型融合视为在黎曼流形上求解弗雷歇均值,并将计算限制在由任务向量张成的低秩子空间中。以期望海塞矩阵为度量,揭示了参数局部曲率与认知不确定性之间的联系。理论分析将融合误差分解为子空间弗雷歇方差与剩余能量,并给出了曲率感知融合在何种条件下能严格优于平坦几何方法的闭式条件。此外,本框架将曲率感知方法与近期谱方法统一为不同几何度量下的子空间弗雷歇均值特例。在八项图像分类任务上对微调后的CLIP-ViT模型进行融合,认知融合在所有三个CLIP-ViT主干网络上均取得更优表现,且在匹配秩条件下,各项指标均优于基线。

原文摘要 · Abstract (English)

Model merging offers a promising avenue for knowledge integration and parallel development without retraining. Yet, existing methods either ignore the geometry of the loss landscape or rely on intractable full-space Hessian approximations. We propose EpiMer, a framework that casts model merging as solving the Fréchet mean on a Riemannian manifold and restricts the computation to a low-rank subspace spanned by the task vectors. With the expected Hessian as the metric, we reveal a connection between local curvature and epistemic uncertainty of the parameters. Our theoretical analysis decomposes the merging error bound into the subspace Fréchet variance and the residual energy, and provides a closed-form characterization of when curvature-aware merging provably outperforms flat-geometry methods. In addition, our framework unifies both curvature-aware methods and recent spectral methods as special cases of the subspace Fréchet mean with different geometric metrics. Merging fine-tuned CLIP-ViT models on eight image classification tasks, Epistemic Merging strictly outperforms the baselines on all three CLIP-ViT backbones at matched rank, improving the across-task average accuracy and worst-task accuracy on every backbone.

模型融合几何视角曲率感知CLIP-ViT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。