400B参数稀疏MoE模型，每令牌激活130亿，训练无崩溃。
Arcee Trinity Large Technical Report
- 采用稀疏专家混合架构，每令牌仅激活部分参数。
- 400B总参数模型在17万亿token上训练，零损失波动。
- 提出新路由策略SMEBU，适合大模型研发与部署场景。
我们发布Arcee Trinity Large的技术报告，该模型为拥有400B总参数、每令牌激活130亿参数的稀疏专家混合（MoE）模型。此外，还介绍了Trinity Nano（6B总参数，10亿激活）和Trinity Mini（26B总参数，30亿激活）。模型采用交错局部与全局注意力、门控注意力、深度缩放沙堡归一化及双曲正切路由等现代架构设计。针对Trinity Large，我们引入一种新的负载均衡策略——软钳制动量专家偏置更新（SMEBU）。所有模型均使用Muon优化器训练，且全程未出现损失突增。Trinity Nano与Trinity Mini在10万亿令牌上预训练，Trinity Large则在17万亿令牌上完成预训练。模型检查点已公开于https://huggingface.co/arcee-ai。
原文摘要 · Abstract (English)
We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models' modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。