按难易程度排序训练数据，让大模型预训练更快更省力。
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
- 用语言难度指标排序数据，从简单到复杂逐步训练。
- 提前阶段提速18%~45%，减少训练步数，提升效率。
- 适合追求高效训练的大模型研究者与工程团队。
课程学习——将训练数据按由易到难顺序组织——在多个机器学习领域提升了训练效率，但在大语言模型预训练中仍研究不足。本文首次系统性探究了大语言模型预训练中的课程学习，共训练超过200个模型，使用最多达100B tokens的数据，采用三种策略：基础课程学习、基于节奏的采样和交错课程，并以六种涵盖语言学与信息论性质的难度指标为指导。在三种真实场景（数据有限、数据无限、持续训练）下评估八个基准任务表现。实验表明，课程学习在早期和中期训练阶段均能显著加速收敛，使达到基线性能所需的训练步数减少18%~45%。作为标准随机采样的预热策略使用时，可带来高达3.5%的持续性能提升。我们识别出压缩比、词汇多样性（MTLD）和可读性（Flesch Reading Ease）是最有效的难度信号。结果表明，数据排序——与现有数据选择方法正交——为更高效的大型语言模型预训练提供了一种实用机制。
原文摘要 · Abstract (English)
Curriculum learning-organizing training data from easy to hard-has improved efficiency across machine learning domains, yet remains underexplored for language model pretraining. We present the first systematic investigation of curriculum learning in LLM pretraining, with over 200 models trained on up to 100B tokens across three strategies: vanilla curriculum learning, pacing-based sampling, and interleaved curricula, guided by six difficulty metrics spanning linguistic and information-theoretic properties. We evaluate performance on eight benchmarks under three realistic scenarios: limited data, unlimited data, and continual training. Our experiments show that curriculum learning consistently accelerates convergence in early and mid-training phases,reducing training steps by $18-45\%$ to reach baseline performance. When applied as a warmup strategy before standard random sampling, curriculum learning yields sustained improvements up to $3.5\%$. We identify compression ratio, lexical diversity (MTLD), and readability (Flesch Reading Ease) as the most effective difficulty signals. Our findings demonstrate that data ordering-orthogonal to existing data selection methods-provides a practical mechanism for more efficient LLM pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。