通过可解释指标预测模型合并成功率，揭示梯度对齐是核心关键。
Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success
- 基于梯度距离等可解释指标，用线性优化预测合并效果。
- 不同方法成功因素差异大，但梯度对齐始终是最重要信号。
- 适合想提升模型合并效果的研究者和工程实践者。
模型合并能整合独立微调模型的知识，但其成功因素仍不明确。我们提出一种与架构无关的框架，发现合并效果不仅取决于模型本身，还受合并方法和任务配对影响。通过在一组可解释的成对指标（如梯度 L_2 距离）上进行 L1 正则化线性优化，我们识别出与合并后归一化准确率相关的关键特性。五种合并方法中，成功驱动因素存在显著差异（平均前5项指标重合率64.0%，符号一致率79.3%），其中 TIES 方法表现出独特“指纹”，偏离普遍规律。但梯度对齐始终是最根本的兼容性信号。这些发现为理解合并能力提供了诊断基础，并推动面向合并的微调策略发展。
原文摘要 · Abstract (English)
Model merging combines knowledge from separately fine-tuned models, yet the factors driving its success remain poorly understood. While recent work treats mergeability as an intrinsic property of the models, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using L1-regularized linear optimization over a set of interpretable pairwise metrics (e.g., gradient L_2 distance), we uncover properties correlating with post-merge normalized accuracy across five merging methods. We find architecture- and method-specific variation in success drivers (64.0% average top-5 metric overlap; 79.3% sign agreement), with certain methods, notably TIES, exhibiting distinct ``fingerprints'' that diverge from the broader consensus. Crucially, however, gradient alignment metrics consistently emerge as the most fundamental signals of compatibility. These findings provide a diagnostic foundation for understanding mergeability and motivate future merge-aware fine-tuning strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。