24亿参数模型用1.08万亿词训练，性能媲美大厂顶尖模型
YuLan-Mini: An Open Data-efficient Language Model
- 构建精细化数据流水线，融合清洗与调度策略提升效率
- 在1.08万亿词上训练，性能达同类模型顶尖水平
- 开源全流程数据配置，适合研究高效训练的团队参考
大规模语言模型的有效预训练面临资源消耗巨大和流程复杂的挑战。本文详细报告了YuLan-Mini的构建过程：一个拥有24.2亿参数的高性能基础模型，在同规模模型中达到顶级表现。其预训练方法通过三项关键技术实现训练效率提升：精细设计的数据流水线结合数据清洗与调度策略；稳健的优化方法缓解训练不稳定性；有效的退火机制融入目标数据选择与长上下文训练。值得注意的是，该模型仅使用1.08万亿词进行训练，性能即媲美需大量数据支持的行业领先模型。为便于复现，我们公开了每个训练阶段的数据组成详情。项目信息见：https://github.com/RUC-GSAI/YuLan-Mini。
原文摘要 · Abstract (English)
Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved. This paper presents a detailed technical report on YuLan-Mini, a highly capable base model with 2.42B parameters that achieves top-tier performance among models of similar parameter scale. Our pre-training approach focuses on enhancing training efficacy through three key technical contributions: an elaborate data pipeline combines data cleaning with data schedule strategies, a robust optimization method to mitigate training instability, and an effective annealing approach that incorporates targeted data selection and long context training. Remarkably, YuLan-Mini, trained on 1.08T tokens, achieves performance comparable to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of the data composition for each training phase. Project details can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。