优化大模型微调数据比例，让效果接近手动调优且更省力。
Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models
- 把数据混合看作优化问题，用缩放定律算出最佳权重。
- 实测平均各领域损失仅比网格搜索最优值高0.66%。
- 可直接重加权常用数据集，提升模型性能，适合快速迭代。
优化大语言模型监督微调（SFT）的数据混合比例对构建通用模型至关重要，但该方向仍研究不足。本文将数据混合建模为优化问题，提出一种新方法以最小化验证损失。通过参数化损失函数，结合微调缩放定律，利用小规模数据混合实验拟合参数并推导最优权重。数学证明与实验证明显示，该算法在所有领域均表现优异，整体与单领域性能俱佳。控制实验表明，使用该方法训练的模型性能与网格搜索确定的最优权重相当，平均各领域损失仅高出0.66%。此外，用该方法重新加权主流SFT数据集，能同时降低验证损失并提升下游任务表现。最后，讨论了方法在特定领域模型数据选择中的推广潜力，并为微调策略提供新洞见。
原文摘要 · Abstract (English)
Optimizing data mixtures for supervised fine-tuning (SFT) of large language models (LLMs) is critical for developing general-purpose models, yet this area remains underexplored. In this paper, we frame data mixing as an optimization problem and introduce a novel method designed to minimize validation loss. Our approach parametrizes the loss by modeling effective data transferred and leveraging scaling laws for fine-tuning. By experimenting with various small-scale data mixtures, we fit these parameters and derive the optimal weights. We provide both mathematical proofs and empirical results demonstrating that our algorithm achieves excellent overall and individual performance across all domains. Through controlled experiments, we show that models trained with our optimized weights perform on par with those using optimal weights determined via grid search, with per-domain loss only 0.66% higher than the best domain loss from grid search on average. Additionally, we show that reweighting popular SFT datasets using our method improves both validation loss and downstream performance. Finally, we discuss how our method can generalize to guide data selection for domain-specific models and provide insights into SFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。