200亿参数模型每令牌仅激活20亿，高效支持长上下文物理AI应用。
TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI
- 动态量化路由实现每令牌自适应专家分配，控制计算开销。
- 128K预训练+512K推理扩展，性能超Qwen3-8B基线模型。
- 适合对延迟敏感的物理智能场景，兼顾效率与长序列处理能力。
我们提出Turing-20B-A2B，一个200亿参数的专家混合语言模型，每令牌激活约20亿参数，专为长上下文和低延迟的物理智能应用设计。模型采用量化路由的动态top-k配置，实现令牌自适应专家分配，保持专家负载均衡并控制平均计算预算。部署时，对提示预填充应用容量约束路由以提升执行规律性和效率，同时在预训练中保留无丢弃路由。模型还采用混合注意力架构，结合闪电注意力与少量全注意力层，实现高效长上下文建模。通过渐进式三阶段预训练，模型扩展至原生128K上下文长度，并在推理时借助YaRN进一步扩展至512K。尽管活跃参数预算紧凑，其基础模型阶段整体通用能力已超越Qwen3-8B Base，接近Qwen3.5-9B Base，同时保持优异的长上下文表现和有利的预填充延迟扩展特性。
原文摘要 · Abstract (English)
We present Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates approximately 2B parameters per token, designed for long-context and latency-sensitive physical AI applications. The model adopts Quantile Routing in a dynamic top-k configuration, enabling token-adaptive expert allocation while maintaining balanced expert utilization and a controlled average compute budget. During deployment, we further apply capacity-constrained routing to prompt prefill for more regular and efficient expert execution, while retaining dropless routing during pretraining. Turing-20B-A2B also employs a hybrid attention architecture that combines Lightning Attention with a small number of full-attention layers for efficient long-context modeling. The model is pretrained with a progressive three-stage curriculum and extended to a native context length of 128K through continued pretraining, with further inference-time extension to 512K using YaRN. Despite its compact active-parameter budget, Turing-20B-A2B achieves, at the base-model stage, overall general capability exceeding Qwen3-8B Base and approaching Qwen3.5-9B Base, while maintaining strong long-context performance and favorable prefill-latency scaling. These results demonstrate an effective balance among model capability, long-context scalability, and practical inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。