通过模型亲缘关系优化大模型合并，提升性能并避免退化。
Exploring Model Kinship for Merging Large Language Models
- 引入模型亲缘度概念，衡量大模型间的相似性。
- 基于亲缘度的贪心合并策略显著提升基准测试表现。
- 适合关注模型迭代与高效合并的研究者和开发者。
模型合并已成为增强大语言模型（LLMs）能力与效率的关键技术。开源社区通过迭代合并现有模型推动模型演进，但对合并带来的收益及其内在机制仍缺乏系统理解。本文将模型演化类比为生物进化，提出‘模型亲缘度’——即大模型间相似性或关联程度的概念。通过全面实证分析，我们发现模型亲缘度与合并后的性能提升密切相关，可作为候选模型选择的有效依据。基于此，我们提出一种新策略：基于模型亲缘度的Top-k贪心合并，该方法在保持持续合并的同时，有效缓解由局部最优导致的性能下降，从而实现更高效的模型演化。代码已公开于https://github.com/zjunlp/ModelKinship。
原文摘要 · Abstract (English)
Model merging has emerged as a key technique for enhancing the capabilities and efficiency of Large Language Models (LLMs). The open-source community has driven model evolution by iteratively merging existing models, yet a principled understanding of the gains and underlying factors in model merging remains limited. In this work, we study model evolution through iterative merging, drawing an analogy to biological evolution, and introduce the concept of model kinship, the degree of similarity or relatedness between LLMs. Through comprehensive empirical analysis, we show that model kinship is closely linked to the performance improvements achieved by merging, providing a useful criterion for selecting candidate models. Building on this insight, we propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can improve benchmark performance. Specifically, we discover that incorporating model kinship as a guiding criterion enables continuous merging while mitigating performance degradation caused by local optima, thereby facilitating more effective model evolution. Code is available at https://github.com/zjunlp/ModelKinship.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。