142B参数中仅激活140亿，实现顶尖模型性能且降低成本。
dots.llm1 Technical Report
- 只激活140亿参数，高效扩展语言模型规模。
- 在11.2万亿高质量文本上训练，性能媲美Qwen2.5-72B。
- 开源每万亿token训练节点，助力大模型研究。
混合专家（Mixture of Experts, MoE）模型通过仅激活部分参数实现语言模型的高效扩展。本文提出 dots.llm1，一个大规模MoE模型，在总计1420亿参数中激活140亿，性能达到当前顶尖水平，同时降低训练与推理开销。该模型在11.2万亿高质量文本上完成预训练，并经过后训练充分释放能力，未使用任何合成数据。为促进研究，我们公开每个万亿级训练步的中间检查点，揭示大模型的学习动态。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this report, we present dots.llm1, a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on 11.2T high-quality tokens and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。