通过预测损失动态选难样本，加速大模型训练收敛。
Predictive Batch Scheduling: Accelerating Language Model Training Through Loss-Aware Sample Prioritization
- 用四个词元特征在线建模样本难度，无需复杂计算。
- 实验显示训练速度提升6-13%，损失相关性从0.14升至0.44。
- 适合追求训练效率的模型开发者，尤其适用于大规模语言模型。
我们提出预测性批处理调度（Predictive Batch Scheduling, PBS），一种新型训练优化方法，通过在批处理构建阶段动态优先处理高损失样本，加速语言模型收敛。与需预定义难度指标的课程学习或依赖昂贵逐样本损失追踪的困难样本挖掘不同，PBS采用轻量级线性预测器，基于静态词元级特征在线估计样本难度。该预测器仅使用四项简单特征：词元频率、序列长度、词汇多样性及稀有词元比例，即可实现与实际损失0.44的相关性。在1.3亿参数Transformer模型上的实验表明，PBS在多个训练检查点上使评估损失收敛速度提升6%-13%；预测相关性在10,000个训练步内由0.14提升至0.44。结果验证了词元频率统计蕴含有效难度信息，支持低开销的高效课程学习。
原文摘要 · Abstract (English)
We introduce Predictive Batch Scheduling (PBS), a novel training optimization technique that accelerates language model convergence by dynamically prioritizing high-loss samples during batch construction. Unlike curriculum learning approaches that require predefined difficulty metrics or hard example mining methods that demand expensive per-sample loss tracking, PBS employs a lightweight linear predictor trained online to estimate sample difficulty from static token-level features. Our predictor achieves 0.44 correlation with actual loss using only four simple features: token frequency, sequence length, vocabulary diversity, and rare token ratio. Experiments on a 130M parameter transformer demonstrate that PBS achieves 6-13\% faster convergence measured by evaluation loss across training checkpoints, with the predictor's correlation improving from 0.14 to 0.44 over 10,000 training steps. These results validate that token frequency statistics encode meaningful information about sample difficulty, enabling effective curriculum learning with negligible computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。