在昇腾超算上实现万亿参数模型全参数微调，提升训练效率并专精运筹优化任务。
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
- 构建分层优化框架，贯通模型并行、通信计算协同与底层内核执行。
- 达到34.22%模型浮点利用率，较开源基线提升2.93倍，训练稳定可靠。
- 专用于运筹优化的微调数据集含1万条高质量样本，性能超越GPT-5.4-Mini。
万亿参数级MoE模型的全参数后训练带来严峻的系统挑战，包括内存压力大、通信重叠差、内核执行效率低。本文以昇腾NPU超算平台为基础，针对DeepSeek-V4模型族开展端到端优化实践。提出涵盖模型级并行、计算-通信协同调度与底层内核优化的分层框架，使系统实现34.22%的模型浮点利用率（MFU），相较开源基线提升2.93倍，同时保持训练稳定性。基于此优化基础设施，构建面向复杂运筹优化（OR）任务的持续预训练（CPT）与监督微调（SFT）流程。使用DeepSeek-V4-Flash开发领域专用数据管道，融合领域资源与求解器验证的合成优化文档，生成包含10,000条高质量样本的数据集，覆盖四类任务与三种问题表示形式。所训练的专用模型在零样本测试中达到71.81%的Pass@1平均得分，优于GPT-5.4-Mini和原DeepSeek-V4-Flash模型，分别高出3.98和11.27个百分点。整体工作展示了从昇腾平台高效训练万亿参数模型，到构建求解器驱动的数学建模专用模型的全栈路径，推动复杂推理前沿模型系统的发展。
原文摘要 · Abstract (English)
Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficient kernel execution. While most large-scale LLM training systems are built around GPU-based clusters, this report presents an end-to-end optimization practice on the Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, we develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution. The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe while maintaining training stability. Building on this optimized infrastructure, we further establish a CPT and SFT workflow for complex Operations Research (OR) tasks. We refer to the integrated framework as SLAI T-Rex. Using DeepSeek-V4-Flash, we develop OR-oriented CPT and SFT data pipelines that combine collected domain resources with solver-verified synthetic optimization documents. The resulting dataset contains 10K high-quality SFT samples spanning four task categories and three problem representations. The specialized model achieves the highest average zero-shot Pass@1 score among the evaluated models, reaching 71.81% and outperforming GPT-5.4-Mini and the base DeepSeek-V4-Flash model by 3.98 and 11.27 percentage points, respectively. Overall, this work demonstrates a full-stack pathway from efficient trillion-parameter model post-training on Ascend infra to domain-specialized Flash models for solver-grounded mathematical modeling, advancing frontier-model systems for complex reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。