让大模型在预训练中始终用自己偏好的数据，提升效果
Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
- 用困惑度差异衡量样本难易，动态分配数据阶段
- 30亿模型在1万亿词上训练，多任务准确率提升超8.1%
- 适合想优化预训练策略的研究者和工程师
大语言模型通常在整个预训练过程中使用一致的数据分布。然而，随着模型能力的提升，其对数据的偏好也会动态变化，因此应在不同训练阶段使用不同数据。为此，我们提出基于困惑度差异（PD）的偏好课程学习框架（PDPC），始终感知并使用模型偏好的数据进行训练以持续提升性能。首先，引入PD指标量化样本对弱模型与强模型的难度差异：高PD样本对弱模型更难，更适合安排在预训练后期。其次，设计偏好函数以近似和预测模型在任意训练步骤的数据偏好，实现数据集的离线排序，保障训练连续性。在13亿和30亿参数模型上的实验表明，PDPC显著优于基线方法。值得注意的是，30亿模型在1万亿词数据上训练后，在MMLU和CMMLU上的平均准确率提升超过8.1%。
原文摘要 · Abstract (English)
Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that its data preferences dynamically change, indicating the need for pretraining with different data at various training stages. To achieve it, we propose the Perplexity Difference (PD) based Preference Curriculum learning (PDPC) framework, which always perceives and uses the data preferred by LLMs to train and boost them. First, we introduce the PD metric to quantify the difference in how challenging a sample is for weak versus strong models. Samples with high PD are more challenging for weak models to learn and are more suitable to be arranged in the later stage of pretraining. Second, we propose the preference function to approximate and predict the data preference of the LLM at any training step, so as to complete the arrangement of the dataset offline and ensure continuous training without interruption. Experimental results on 1.3B and 3B models demonstrate that PDPC significantly surpasses baselines. Notably, the 3B model trained on 1T tokens achieves an increased average accuracy of over 8.1% across MMLU and CMMLU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。