120B参数混合专家模型，推理速度超GPT和通义千问。
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- 融合Mamba与注意力机制的混合专家架构，提升计算效率。
- 支持百万级上下文，推理吞吐量最高达GPT的2.2倍。
- 开源全链路模型，适合需要高效推理的开发者使用。
我们介绍了Nemotron 3 Super的预训练、后训练及量化过程，这是一个拥有1200亿（活跃120亿）参数的混合Mamba-注意力混合专家模型。Nemotron 3 Super是Nemotron 3系列中首个在NVFP4下进行预训练、采用新型混合专家架构LatentMoE（优化每浮点运算精度与每参数精度）、并集成MTP层以实现原生推测解码加速的模型。模型在25万亿个标记上进行预训练，随后通过监督微调（SFT）和强化学习（RL）进行后训练。最终模型支持高达100万的上下文长度，在常见基准测试中表现相当，同时相比GPT-OSS-120B和Qwen3.5-122B，推理吞吐量分别提升最多达2.2倍和7.5倍。Nemotron 3 Super的数据集以及基础、后训练和量化版本模型检查点均已开源至HuggingFace。
原文摘要 · Abstract (English)
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。