用模型训练影响度排序数据，让预训练更高效
Influence-driven Curriculum Learning for Pre-training on Limited Data
- 按每个样本对模型输出的影响大小排序训练数据
- 在多个基准上性能比随机顺序训练高出10个百分点以上
- 适合做语言模型预训练优化的研究者和工程师
课程学习是一种按示例难度（如从简单到复杂文档）顺序训练模型的技术，但在语言模型预训练中表现有限。本文探究：若将传统依赖人工的难度指标替换为更贴近模型训练过程实际难度的指标，课程学习是否更具竞争力。具体而言，我们尝试按训练数据影响度（training data influence）对训练样本排序，该分数用于估计单个样本对模型输出的影响。实验表明，基于此课程学习训练的模型在多个基准测试中性能优于随机顺序训练的模型，提升超过10个百分点，验证了只要采用更以模型为中心的难度定义，课程学习确实能有效提升语言模型预训练效果。
原文摘要 · Abstract (English)
Curriculum learning, a training technique where data is presented to the model in order of example difficulty (e.g., from simpler to more complex documents), has shown limited success for pre-training language models. In this work, we investigate whether curriculum learning becomes competitive if we replace conventional human-centered difficulty metrics with one that more closely corresponds to example difficulty as observed during model training. Specifically, we experiment with sorting training examples by their \textit{training data influence}, a score which estimates the effect of individual training examples on the model's output. Models trained on our curricula are able to outperform ones trained in random order by over 10 percentage points in benchmarks, confirming that curriculum learning is beneficial for language model pre-training, as long as a more model-centric notion of difficulty is adopted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。