无需数据即可高效合并模型，提升多任务性能
Model Merging via Data-Free Covariance Estimation
- 通过差分矩阵直接估算协方差，免去数据依赖
- 在86M到7B参数模型上均超越现有无数据方法
- 适合资源受限场景下的模型集成应用
模型合并可低成本融合多个模型以继承其能力。尽管部分合并方法接近多任务训练性能，但多为启发式设计且缺乏理论依据。一种合理替代方案是将模型合并建模为逐层优化问题，直接最小化任务间干扰。然而该方法需从数据中估计每层协方差矩阵，当无法获取数据时难以实施。相比之下，许多启发式方法无需辅助数据，更具实用性。本文重新审视干扰最小化框架，证明在特定条件下，协方差矩阵可直接由差分矩阵估算，从而无需数据并降低计算开销。我们在视觉与语言基准上验证了该方法，涵盖从86M到7B参数的模型，性能优于此前无数据最优方法。
原文摘要 · Abstract (English)
Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from 86M parameters to 7B parameters, outperforming previous data-free state-of-the-art merging methods
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。