309B参数模型用15B激活量实现强推理，推理速度提升2.6倍。
MiMo-V2-Flash Technical Report
- 混合注意力架构结合滑动窗口与全局注意力，128令牌滑窗比例5:1。
- 27万亿词预训练，上下文长度达256k，推理速度最高快2.6倍。
- 开源权重，适合追求高效推理的开发者和研究者使用。
我们提出 MiMo-V2-Flash，一个拥有3090亿总参数、仅激活150亿参数的专家混合（MoE）模型，专为快速且强大的推理与代理能力设计。该模型采用混合注意力结构，将滑动窗口注意力（SWA）与全局注意力交替使用，滑动窗口为128令牌，混合比例为5:1。模型在27万亿令牌上进行多标记预测（MTP）预训练，原生支持32k上下文长度，并扩展至256k。为高效扩展后训练计算，提出新型多教师在线蒸馏（MOPD）范式：领域专用教师（如通过大规模强化学习训练）提供密集的逐标记奖励，使学生模型完全掌握教师知识。尽管仅使用DeepSeek-V3.2和Kimi-K2约一半和三分之一的总参数，其性能仍可比肩顶尖开源模型。推理阶段，通过将MTP重用于推测解码的草稿模型，利用三层MTP实现最高3.6的接受长度和2.6倍的解码加速。模型权重及三层MTP权重已开源，以促进开放研究与社区协作。
原文摘要 · Abstract (English)
We present MiMo-V2-Flash, a Mixture-of-Experts (MoE) model with 309B total parameters and 15B active parameters, designed for fast, strong reasoning and agentic capabilities. MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention, with a 128-token sliding window under a 5:1 hybrid ratio. The model is pre-trained on 27 trillion tokens with Multi-Token Prediction (MTP), employing a native 32k context length and subsequently extended to 256k. To efficiently scale post-training compute, MiMo-V2-Flash introduces a novel Multi-Teacher On-Policy Distillation (MOPD) paradigm. In this framework, domain-specialized teachers (e.g., trained via large-scale reinforcement learning) provide dense and token-level reward, enabling the student model to perfectly master teacher expertise. MiMo-V2-Flash rivals top-tier open-weight models such as DeepSeek-V3.2 and Kimi-K2, despite using only 1/2 and 1/3 of their total parameters, respectively. During inference, by repurposing MTP as a draft model for speculative decoding, MiMo-V2-Flash achieves up to 3.6 acceptance length and 2.6x decoding speedup with three MTP layers. We open-source both the model weights and the three-layer MTP weights to foster open research and community collaboration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。