用模型合并权重高效优化大模型训练数据配比，省时省力
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
- 把模型合并权重当性能代理，快速找最佳数据混合比例
- 8B和16B模型上表现媲美人工调参，搜索成本大幅降低
- 适合大规模模型训练中的自动化数据配比优化场景
优化数据混合比例对释放大语言模型潜力至关重要，但传统方法依赖启发式尝试或昂贵的代理训练，计算成本高。为此，我们提出MergeMix，通过将模型合并权重作为高保真、低成本的性能代理，高效确定最优数据混合比例。仅需在少量样本上训练领域专家，并基于下游基准优化其合并权重，即可实现无需全量训练的数据混合优化。在8B和16B参数模型上的大量实验表明，MergeMix性能达到甚至超过人工穷举调参，同时显著降低搜索成本。此外，该方法具有高秩一致性（Spearman ρ > 0.9）和强跨规模可迁移性，提供了一种可扩展的自动化数据混合优化方案。
原文摘要 · Abstract (English)
Optimizing data mixtures is essential for unlocking the full potential of large language models (LLMs), yet identifying the optimal composition remains computationally prohibitive due to reliance on heuristic trials or expensive proxy training. To address this, we introduce \textbf{MergeMix}, a novel approach that efficiently determines optimal data mixing ratios by repurposing model merging weights as a high-fidelity, low-cost performance proxy. By training domain-specific experts on minimal tokens and optimizing their merging weights against downstream benchmarks, MergeMix effectively optimizes the performance of data mixtures without incurring the cost of full-scale training. Extensive experiments on models with 8B and 16B parameters validate that MergeMix achieves performance comparable to or surpassing exhaustive manual tuning while drastically reducing search costs. Furthermore, MergeMix exhibits high rank consistency (Spearman $ρ> 0.9$) and strong cross-scale transferability, offering a scalable, automated solution for data mixture optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。