用弗雷切特平均统一模型合并几何，提升对称性敏感度。
Generalizing the Geometry of Model Merging Through Frechet Averages
- 基于测地距离的弗雷切特平均，实现对称不变的模型合并。
- 在低秩适配器（LoRA）上验证，新方法优于现有合并策略。
- 适合研究模型融合、架构对称性问题的工程师与学者。
模型合并旨在不进行额外训练的情况下将多个模型融合为一个。朴素的参数空间平均在面对架构对称性时表现脆弱，因其未考虑对称性带来的几何结构。本文表明，不仅几何本身，连平均过程也必须具备对称性不变性，才能实现真正的对称感知合并。为此，我们提出一种通用方案：将合并视为弗雷切特平均，即在适当流形上选择使测地距离总和最小的参数。核心设计在于几何选择——包括度量、流形及距离近似方式，决定两个模型“接近”的定义。我们证明，在简化假设下，弗雷切特平均包含费舍尔合并。进一步，针对低秩适配器（LoRA）的对称性诱导出商流形几何，分析了现有合并方法的局限性，提出实用算法，并与常见方法对比验证其优势。
原文摘要 · Abstract (English)
Model merging aims to combine multiple models into one without additional training. Naïve parameter-space averaging can be fragile under architectural symmetries, as their geometry does not take them into account. In this work we show that not only the geometry, but also the averaging procedure itself, must be symmetry-invariant to achieve symmetry-aware merges. Consequently, we propose a general solution: merging as Fréchet averaging, i.e., selecting parameters that minimize a sum of geodesic distances on an appropriate manifold. In this view, the key design choice is the overall geometry, i.e., the choice of metric, manifold, and distance approximation, that determines what it means for two models to be "close". We show that Fréchet averaging, combined with simplifying assumptions, contains Fisher merging. Building on this, we examine the particular case of low-rank adapters (LoRA), whose symmetries induce a distinct geometry: that of a quotient manifold. We outline the limitations of current LoRA merging methods, propose a practical algorithm for this setting, and show how they compare with other commonly used approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。