通过渐进式加层匹配数据难度，提升大模型预训练效率与泛化能力。
Curriculum-Guided Layer Scaling for Language Model Pretraining
- 训练中逐步增加模型层数，同步提升数据难度
- 在100M和1.2B参数规模下均提升问答任务性能
- 适合追求高效预训练的模型研发人员
随着大语言模型预训练成本上升，提升训练阶段学习效率成为关键。受人类认知发展启发，我们提出课程引导的层渐进扩展（CGLS）框架，通过渐进式堆叠层数，使模型成长与数据难度同步提升。在1亿参数规模下，使用从合成短篇故事到通用网页数据的课程，CGLS在PIQA和ARC问答基准上优于基线方法；在12亿参数规模下，利用基于DistilBERT的分类器对DataComp-LM语料库进行分层，从通用文本逐步过渡到高技术性内容。结果表明，模型深度与样本难度同步递增可显著提升泛化能力和零样本性能。研究验证了渐进式堆叠的有效性，为知识密集型与推理类任务提供了简单高效的预训练策略。
原文摘要 · Abstract (English)
As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core training stage. Motivated by cognitive development, where humans gradually build knowledge as their brains mature, we propose Curriculum-Guided Layer Scaling (CGLS), a framework for compute-efficient pretraining that synchronizes increasing data difficulty with model growth through progressive layer stacking (i.e. gradually adding layers during training). At the 100M parameter scale, using a curriculum transitioning from synthetic short stories to general web data, CGLS outperforms baseline methods on the question-answering benchmarks PIQA and ARC. Pretraining at the 1.2B scale, we stratify the DataComp-LM corpus with a DistilBERT-based classifier and progress from general text to highly technical or specialized content. Our results show that progressively increasing model depth alongside sample difficulty leads to better generalization and zero-shot performance on various downstream benchmarks. Altogether, our findings demonstrate that CGLS unlocks the potential of progressive stacking, offering a simple yet effective strategy for improving generalization on knowledge-intensive and reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。