arXiv:2602.04937cs.LGcs.AI2026-02

用模型合并快速预测最优数据混合比例,省去大量训练成本。

Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization

  • 通过加权合并各领域专家模型,模拟不同数据混合效果。
  • 在14个基准上验证,合并模型与真实训练结果相关性高达0.92。
  • 适合需要高效调参的多模态大模型研发团队使用。

选择最佳数据混合比例对多模态大语言模型的监督微调至关重要。然而,由于组合搜索空间庞大且单次训练成本高昂,确定多个领域特定数据集的最佳混合权重仍是一个主要瓶颈,即所谓的数据混合优化(DMO)问题。另一方面,模型合并通过参数插值统一领域专家模型,效率高,只需每个领域一次训练,但常导致次优结果。本文结合两者优势,研究将模型合并作为高效估计不同数据混合性能的策略。我们训练领域特定的多模态专家模型,并评估其加权参数空间组合,以预测对应数据混合的实际效果。在14个多模态基准上进行广泛实验,实证表明合并代理模型与实际数据混合训练模型之间具有高度秩相关性(平均0.92)。该方法将最优混合搜索与资源密集型训练过程解耦,为复杂混合权重空间提供可扩展、高效的探索方案。代码已公开于https://github.com/BerasiDavide/mLLMs_merging_4_DMO。

原文摘要 · Abstract (English)

Selecting the best data mixture is critical for successful Supervised Fine-Tuning (SFT) of Multimodal Large Language Models. However, determining the optimal mixture weights across multiple domain-specific datasets remains a significant bottleneck due to the combinatorial search space and the high cost associated with even a single training run. This is the so-called Data Mixture Optimization (DMO) problem. On the other hand, model merging unifies domain-specific experts through parameter interpolation. This strategy is efficient, as it only requires a single training run per domain, yet oftentimes leads to suboptimal models. In this work, we take the best of both worlds, studying model merging as an efficient strategy for estimating the performance of different data mixtures. We train domain-specific multimodal experts and evaluate their weighted parameter-space combinations to estimate the efficacy of corresponding data mixtures. We conduct extensive experiments on 14 multimodal benchmarks, and empirically demonstrate that the merged proxy models exhibit a high rank correlation with models trained on actual data mixtures. This decouples the search for optimal mixtures from the resource-intensive training process, thereby providing a scalable and efficient strategy for navigating the complex landscape of mixture weights. Code is publicly available at https://github.com/BerasiDavide/mLLMs_merging_4_DMO.

多模态模型合并数据混合高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。