1.7亿参数小模型通过海量数据训练，性能超越多个同类模型。
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- 用11万亿词分阶段训练，融合数学、代码和指令数据提升能力
- 在数学与指令任务上超越Qwen2.5-1.5B和Llama3.2-1B等模型
- 开源模型及自建高质量数据集，助力小模型研究与应用
尽管大语言模型在人工智能诸多应用中取得突破，但其庞大体积导致计算成本高，在资源受限环境下部署困难。本文介绍SmolLM2的开发，这是一个参数量为1.7亿的先进小型语言模型。为实现优异性能，我们使用约11万亿个令牌的数据，通过多阶段训练过程，混合网络文本与专用数学、代码及指令遵循数据进行过训练。此外，在发现现有数据集规模过小或质量不佳时，我们引入了新的专用数据集（FineMath、Stack-Edu和SmolTalk）。为指导设计决策，我们进行了小规模消融实验，并采用人工优化方式根据前一阶段表现调整各阶段数据混合比例。最终结果表明，SmolLM2在多项任务上优于其他近期小型语言模型，包括Qwen2.5-1.5B和Llama3.2-1B。为促进小型语言模型的研究与应用，我们公开发布SmolLM2及其在整个项目中构建的所有数据集。
原文摘要 · Abstract (English)
While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。