用强化学习实现跨架构模型合并，支持多任务定制化。
HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models
- 将模型合并建模为强化学习任务，自动探索不同架构组合。
- 在翻译、推理、代码生成任务上超越现有方法，性能提升显著。
- 支持多目标优化，根据需求生成个性化合并方案，适合多场景应用。
模型合并是一种将多个大型预训练模型融合为单一模型的技术，可提升性能并增强任务适应性，因其无需原始训练数据和再训练过程而受到关注。然而，现有方法主要局限于相同架构的参数空间合并，跨架构合并因搜索空间庞大且层兼容性挑战而进展缓慢。本文首次将架构空间合并建模为强化学习问题，通过离线采样权重向量训练策略与价值网络，用于在线优化合并策略。同时引入多目标优化框架，学习最优模型的帕累托前沿，以满足用户多样化任务偏好，提供定制化合并建议。在文本翻译、数学推理和代码生成等多个任务上的实验验证了该框架的有效性和优越性。代码将在审稿后公开。
原文摘要 · Abstract (English)
Model merging is a technique that combines multiple large pretrained models into a single model with enhanced performance and broader task adaptability. It has gained popularity in large pretrained model development due to its ability to bypass the need for original training data and further training processes. However, most existing model merging approaches focus solely on exploring the parameter space, merging models with identical architectures. Merging within the architecture space, despite its potential, remains in its early stages due to the vast search space and the challenges of layer compatibility. This paper marks a significant advance toward more flexible and comprehensive model merging techniques by modeling the architecture-space merging process as a reinforcement learning task. We train policy and value networks using offline sampling of weight vectors, which are then employed for the online optimization of merging strategies. Moreover, a multi-objective optimization paradigm is introduced to accommodate users' diverse task preferences, learning the Pareto front of optimal models to offer customized merging suggestions. Experimental results across multiple tasks, including text translation, mathematical reasoning, and code generation, validate the effectiveness and superiority of the proposed framework in model merging. The code will be made publicly available after the review process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。