从零开始训练10亿参数中文大模型，开源全程经验。
Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM
- 用有限算力从头训练中文优先的10亿参数模型
- 在CEVAL和CMMLU上表现优于早期大机构模型
- 公开数据、代码和踩坑记录，适合想自建模型者
Steel-LLM 是一个从零开始构建的中文聚焦语言模型，旨在以有限计算资源打造高质量、开源的模型。2024年3月启动，目标是在大规模数据集上训练一个10亿参数模型，强调透明性与实践分享，助力社区发展。训练以中文数据为主，辅以少量英文数据，弥补现有开源大模型在中文场景下的不足。模型在CEVAL和CMMLU等基准测试中表现优异，超越部分早期大型机构发布的模型。本文系统总结了项目在数据收集、模型设计、训练方法及挑战应对方面的核心贡献，为希望自主开发大模型的研究人员和从业者提供宝贵参考。模型权重与训练脚本已公开于 https://github.com/zhanshijinwat/Steel-LLM。
原文摘要 · Abstract (English)
Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。