arXiv:2604.08510cs.CL2026-04被引 8

语言模型训练时按可预测顺序逐步掌握技能,先学基础再学复合任务。

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

  • 通过设计可组合的任务追踪模型能力出现时机,发现技能按成分顺序逐步涌现。
  • 45组模型对比显示能力出现顺序高度一致(ρ=0.81),复合任务总在组件之后出现。
  • 模型内部表征能预测新任务的训练轨迹,准确率达R²=0.68–0.84。

大语言模型虽能完成复杂任务,但其预训练过程中能力如何逐步形成仍不清晰。现有缩放定律仅说明计算量增加对损失下降的影响,却无法揭示具体技能的习得顺序。为此,我们提出隐式课程假说:预训练遵循一种可组合且可预测的技能习得路径,适用于不同模型和数据混合。通过设计涵盖检索、形态变换、指代消解、逻辑推理与数学计算的简单可组合任务,我们在四类模型(参数量410M–13B)上追踪了达到固定准确率阈值的出现时机。结果发现,45组模型间的技能出现顺序高度一致(ρ=0.81),且复合任务通常在组成它的基础任务之后才出现。此外,模型内部表示中任务功能向量相似者,其训练轨迹也趋同。基于这些任务构建的表示空间,可有效预测未评估的组合任务在整个预训练过程中的表现轨迹(各模型R²=0.68–0.84)。这些结果表明,预训练远比损失曲线所揭示的更有序:技能以可预测的组合方式依次浮现,并可从模型内部读取。

原文摘要 · Abstract (English)

Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent ($ρ= .81$ across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining ($R^2 = .68$-$.84$ across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.

语言模型预训练能力涌现课程假说

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。