通过凸优化找到最优数据混合，显著提升模型性能。
MixMin: Finding Data Mixtures via Convex Minimization
- 将数据混合问题转化为凸优化问题，用梯度法求解。
- 在多个任务上实现1-5%的负对数似然降低，仅增0.2%计算量。
- 混合方案具尺度不变性，适合小模型提升大模型训练。
现代机器学习系统越来越多地融合来自不同来源的数据，例如大规模语言模型的预训练。然而，寻找最优数据混合仍是一个具有挑战性的开放问题。本文将该问题形式化为双层目标：最优混合应使下游任务表现最佳。由于该目标通常不可行，我们观察到当模型类别增大时，双层混合目标变为凸的。为此，我们提出基于梯度的优化方法MixMin，并在语言建模和化学任务上进行测试。MixMin是唯一在所有实验中均改善数据混合的方法。使用小于0.2%额外计算量，对基于82亿个标记的pythia-410M模型进行微调，使PIQA、ARC Easy、SciQ和OpenWebMath上的负对数似然相对下降1-5%。关键发现是，小模型的MixMin混合方案能有效提升大模型训练效果，表明其具有尺度不变性。在将生物活性数据混合用于训练XGBoost模型时，平均精度得分提升0.03-0.15。
原文摘要 · Abstract (English)
Modern machine learning pipelines are increasingly combining and mixing data from diverse and disparate sources, e.g., pre-training large language models. Yet, finding the optimal data mixture is a challenging and open problem. We formalize this data mixing problem as a bi-level objective: the best mixture is the one that would lead to the best model for a downstream objective. Unfortunately, this objective is generally intractable. In this paper, we make the observation that the bi-level data mixing objective becomes convex as our model class becomes larger. We develop and study a gradient-based approach for optimizing this convex objective, which we call MixMin, and test it on language modeling and chemistry tasks. MixMin was the only method that uniformly improved the data mixture in all our experiments. With MixMin, we improved the data mixture using less than 0.2% additional compute for a pythia-410M model trained on 8.2B tokens, resulting between 1-5% relative improvement to negative log likelihood on PIQA, ARC Easy, SciQ, and OpenWebMath. Crucially, we found that MixMin mixtures for smaller models improved training of larger models, suggesting that MixMin mixtures may be scale-invariant. When mixing bioassay data to train an XGBoost model, we saw improvements to average precision scores of 0.03-0.15.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。