用贝叶斯优化自动找模型合并的最佳参数，省时省力效果好。
Multi-Objective Bayesian Optimization for Model Merging
- 把合并参数选择建模为多目标优化问题，用贝叶斯方法高效搜索
- 在12组对比中11次优于随机搜索，尤其对复杂合并方式提升明显
- 不依赖特定合并算子，适合需要多目标权衡的模型融合场景
模型合并通过直接在权重空间整合训练好的模型，提供一种比微调更节省计算资源的替代方案。然而，合并参数的选择困难，因下游评估成本高、无梯度信息且源模型能力可能冲突。本文将参数选择问题建模为黑箱多目标优化，并提出MOBO-Merge框架，采用多目标贝叶斯优化在有限评估预算下逼近帕累托前沿。我们在Qwen3-4B和Llama-3.1-8B上，使用Linear、SLERP、TIES及分块合并算子，在两模型指令-数学与三模型指令-数学-代码设置中进行评估。在保留的基准分区上，MOBO-Merge在12次比较中有11次获得更高的平均超体积。一维线性插值提升较小，但多种TIES、分块及三目标搜索的增益显著。没有统一最优的合并算子：TIES在四组家族设定中表现最佳，而Block-Linear 4x在Llama三模型合并中最强。结果表明，多目标贝叶斯优化是表达性强合并参数化的重要搜索层。
原文摘要 · Abstract (English)
Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning. Selecting merge parameters is nevertheless difficult because downstream evaluations are expensive, gradients are unavailable, and source capabilities can conflict. We formulate merge-parameter selection as a black-box multi-objective optimization problem and introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization to approximate the Pareto front under a limited evaluation budget. We evaluate Qwen3-4B and Llama-3.1-8B in two-model instruction-math and three-model instruction-math-code settings using Linear, SLERP, TIES, and block-wise merge operators. On held-out benchmark partitions, MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 reported comparisons. The gain is small for one-dimensional Linear interpolation but substantially larger for several TIES, block-wise, and three-objective searches. No merge operator is uniformly best: TIES leads in three of four family-setting combinations, whereas Block-Linear 4x is strongest for the Llama three-model merge. These results show that multi-objective Bayesian optimization is valuable as a search layer for expressive merge parameterizations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。