通过筛选子网络和知识蒸馏，让小模型预训练更高效。
Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
- 用进化搜索找优质初始子网络，替代随机初始化。
- 相比同类模型，100B词元下节省1.26倍计算量。
- 适合想低成本训练高性能小模型的研究者。
小语言模型（SLMs）在资源消耗远低于大语言模型（LLMs）的前提下仍能保持优异性能。本文提出一个简单有效的SLM预训练框架，融合三项互补策略：首先，识别出结构稀疏的子网络初始化，其在相同算力预算下表现优于同规模随机初始化模型；其次，采用进化搜索自动发现高质量子网络初始化，为预训练提供更优起点；第三，利用大模型教师进行知识蒸馏，加速训练并提升泛化能力。三者结合使SLM预训练效率显著提升：最佳模型通过进化搜索并以LLM权重初始化，在100B词元预算下验证困惑度与对比的Pythia SLM相当，但计算量仅需其1.26倍；在10B词元预算下则只需5.16倍。代码已公开，为大规模低成本小模型开发提供可复现路径。
原文摘要 · Abstract (English)
Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introduce a simple and effective framework for pretraining SLMs that brings together three complementary ideas. First, we identify structurally sparse sub-network initializations that consistently outperform randomly initialized models of similar size under the same compute budget. Second, we use evolutionary search to automatically discover high-quality sub-network initializations, providing better starting points for pretraining. Third, we apply knowledge distillation from larger teacher models to speed up training and improve generalization. Together, these components make SLM pretraining substantially more efficient: our best model, discovered using evolutionary search and initialized with LLM weights, matches the validation perplexity of a comparable Pythia SLM while requiring 5.16x and 1.26x fewer floating point operations for token budgets of 10B and 100B, respectively. We release all code publicly, offering a practical and reproducible path toward cost-efficient small language model development at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。