为大模型合并超参优化设计低成本替代基准，加速算法研发。
Surrogate Benchmarks for Model Merging Optimization
- 构建双搜索空间与数据集，训练代理模型预测合并性能
- 代理模型可精准预测合并效果，误差低于10%
- 适合研究模型合并算法的开发者快速验证新方法
模型合并技术旨在将多个模型的能力整合到单一模型中。多数合并方法包含超参数，其设置直接影响合并后模型的性能。已有研究显示，通过调优超参数可显著提升合并效果，因此开发面向模型合并的超参数优化算法具有广阔前景。然而，该优化过程计算成本高昂，尤其在大型语言模型（LLMs）合并时更为显著。本文提出用于合并超参数优化的代理基准，实现算法开发与性能比较的低成本验证。我们定义了两个搜索空间，并收集数据样本以构建代理模型，用于根据超参数预测合并模型的性能。实验表明，所提基准能有效预测合并模型表现，且可准确模拟优化算法的行为。
原文摘要 · Abstract (English)
Model merging techniques aim to integrate the abilities of multiple models into a single model. Most model merging techniques have hyperparameters, and their setting affects the performance of the merged model. Because several existing works show that tuning hyperparameters in model merging can enhance the merging outcome, developing hyperparameter optimization algorithms for model merging is a promising direction. However, its optimization process is computationally expensive, particularly in merging LLMs. In this work, we develop surrogate benchmarks for optimization of the merging hyperparameters to realize algorithm development and performance comparison at low cost. We define two search spaces and collect data samples to construct surrogate models to predict the performance of a merged model from a hyperparameter. We demonstrate that our benchmarks can predict the performance of merged models well and simulate optimization algorithm behaviors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。