蒸馏预训练提升模型测试时扩展能力，但削弱上下文学习性能。
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
- 用蒸馏方法预训练，使模型在测试时扩展表现更优。
- 蒸馏后模型的上下文学习能力下降，尤其影响归纳头机制。
- 通过简化模型验证发现核心影响因素，指导未来训练设计。
过去一年中，蒸馏在大语言模型（LLM）预训练中重新受到重视，典型代表是Llama-3.2和Gemma系列模型。尽管蒸馏长期被证明能提升统计建模效果，但其对现代LLM关键范式——如测试时扩展（test-time scaling）和上下文学习（in-context learning）的影响仍不明确。本文有三大贡献：首先，我们发现蒸馏预训练模型展现出显著更强的测试时扩展能力；其次，该优势伴随代价：蒸馏会削弱上下文学习能力，特别是由归纳头（induction heads）所建模的部分；第三，我们构建了一个基于大词频模型（bigram model）的简化实验环境，隔离出影响观察结果的核心因素。最后，基于这些发现，我们为未来的预训练设计提供了实用洞见。
原文摘要 · Abstract (English)
In the past year, distillation has seen a renewed prominence in large language model (LLM) pretraining, exemplified by the Llama-3.2 and Gemma model families. While distillation has historically been shown to improve statistical modeling, its effects on new paradigms that are key to modern LLMs, such as test-time scaling and in-context learning, remain underexplored. In this work, we make three main contributions. First, we show that pretraining with distillation yields models that exhibit remarkably better test-time scaling. Second, we observe that this benefit comes with a trade-off: distillation impairs in-context learning capabilities, particularly the one modeled via induction heads. Third, to demystify these findings, we study distilled pretraining in a sandbox of a bigram model, which helps us isolate the common principal factor behind our observations. Finally, using these insights, we shed light on various design choices for pretraining that should help practitioners going forward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。