用数据专家混合模型优化预训练数据组合，提升语言模型性能。
Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
- 通过数据专家混合模型近似不同数据组合的交叉熵损失。
- 在70M到1B参数模型上，性能显著优于仅用混合比例的基线方法。
- 适合关注预训练数据优化与小样本下游任务的科研人员。
我们提出一种方法，通过混合数据专家（MDE）高效近似每种候选数据混合组合对应的交叉熵损失，将该近似结果作为回归模型的额外特征输入。该回归模型基于少量混合组合的模型损失观测值进行训练。在SlimPajama数据集上，对70M至1B参数的Transformer解码器语言模型进行实验，结果表明，相比仅使用混合比例作为输入特征的回归模型，本方法性能显著更优。结合考虑下游任务数据交叉熵的目标函数后，进一步提升了小样本下游评估表现。此外，我们还提供了理论分析，解释为何数据专家预测的聚合能良好近似数据混合下的模型损失。
原文摘要 · Abstract (English)
We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mixture of Data Experts (MDE). We use this approximation as a source of additional features in a regression model, trained from observations of model loss for a small number of mixtures. Experiments with Transformer decoder-only language models in the range of 70M to 1B parameters on the SlimPajama dataset show that our method achieves significantly better performance than approaches that train regression models using only the mixture rates as input features. Combining this improved optimization method with an objective that takes into account cross-entropy on end task data leads to superior performance on few-shot downstream evaluations. We also provide theoretical insights on why aggregation of data expert predictions can provide good approximations to model losses for data mixtures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。