小模型持续预训练可高效适配教育领域，提升知识与理解能力。
Domain-Adaptive Continued Pre-Training of Small Language Models
- 用125M参数模型在4亿+数据上逐步训练，最终达10亿token
- MMLU提升8.1%，HellaSwag提升7.6%，知识与上下文理解显著增强
- 适合资源有限场景下教育类小模型的定制化优化
小语言模型的持续预训练为计算资源有限条件下的领域适配提供了可行路径。本文在教育领域评估该方法，将其作为从零训练的高效替代方案。采用125M参数模型，在4亿个标记上进行增量训练，随后继续训练至10亿标记。方法包括全面的数据预处理、内存优化的训练配置及基于基准的评估。结果表明，知识密集型任务（MMLU +8.1%）和上下文理解任务（HellaSwag +7.6%）均有显著提升，同时揭示了教育领域专属性的权衡。分析了标记效率、灾难性遗忘缓解策略及扩展规律。研究显示，通过精心设计的预处理与训练方法，即使在资源受限条件下，也能显著提升小模型能力，为小模型在特定领域的适配开辟新路径。
原文摘要 · Abstract (English)
Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources. I've investigated this approach within educational domains, evaluating it as a resource-efficient alternative to training models from scratch. Using a 125M parameter model, I demonstrate significant performance improvements through incremental training on 400 million tokens, followed by further training to reach 1 billion tokens. My approach includes comprehensive data preprocessing, memory-optimized training configurations, and benchmark-based evaluation. Results show notable gains in knowledge-intensive tasks (MMLU +8.1%) and contextual understanding (HellaSwag +7.6%), while revealing educational domain specialization trade-offs. I analyze token efficiency, catastrophic forgetting mitigation strategies, and scaling patterns. My findings suggest that thoughtful preprocessing and training methodologies enable meaningful improvements in language model capabilities even with constrained computational resources, opening pathways for domain-specific adaptation of smaller language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。