DeepSeek-V3用专家混合架构实现强性能，训练稳定且成本低。
DeepSeek-V3 Technical Report
- 采用多头隐式注意力与专家混合结构，提升推理效率。
- 6710亿参数仅激活370亿，训练仅需278.8万张H800 GPU小时。
- 无辅助损失负载均衡，适合追求高效大模型的开发者。
我们提出 DeepSeek-V3，一个拥有6710亿总参数、每令牌激活370亿参数的强健混合专家（MoE）语言模型。为实现高效推理和低成本训练，DeepSeek-V3 采用在 DeepSeek-V2 中经过验证的多头隐式注意力（MLA）和 DeepSeekMoE 架构。此外，该模型首次提出无辅助损失的负载均衡策略，并设置多令牌预测训练目标以增强性能。模型在14.8万亿条多样且高质量的文本上进行预训练，随后经过监督微调和强化学习阶段，充分释放其潜力。全面评估显示，DeepSeek-V3 超越其他开源模型，性能媲美顶尖闭源模型。尽管表现优异，其完整训练仅需2.788百万张H800 GPU小时，且训练过程极为稳定：全程未出现不可恢复的损失突增或回滚操作。模型检查点已公开于 https://github.com/deepseek-ai/DeepSeek-V3。
原文摘要 · Abstract (English)
We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。