MiniMax-M1用闪电注意力实现高效长文本推理,支持百万级上下文。
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- 混合专家架构+闪电注意力,动态激活459亿参数处理每令牌。
- 支持100万词上下文长度,测试时计算可高效扩展。
- 适合复杂编程、长文本分析任务,开源模型可用作研究基准。
我们提出MiniMax-M1,首个开源权重的大规模混合注意力推理模型。该模型基于前序的MiniMax-Text-01,共4560亿参数,每令牌激活459亿参数。M1原生支持100万词上下文长度,是DeepSeek R1的8倍。其闪电注意力机制使测试时计算可高效扩展,特别适用于需要长输入和深度思考的任务。模型通过大规模强化学习(RL)在沙盒环境与真实软件工程场景中训练。为提升训练效率,我们提出CISPO算法,通过裁剪重要性采样权重而非更新令牌,优于其他主流RL方法。结合混合注意力与CISPO,仅用512块H800 GPU,在三周内完成完整训练,租赁成本仅53.47万美元。我们发布两个版本:4万和8万思维预算,其中4万版本为8万训练的中间阶段。标准基准测试显示,模型性能可比肩或超越DeepSeek-R1与Qwen3-235B,尤其在复杂软件工程、工具调用和长上下文任务中表现突出。代码与模型已公开于https://github.com/MiniMax-AI/MiniMax-M1。
原文摘要 · Abstract (English)
We introduce MiniMax-M1, the world's first open-weight, large-scale hybrid-attention reasoning model. MiniMax-M1 is powered by a hybrid Mixture-of-Experts (MoE) architecture combined with a lightning attention mechanism. The model is developed based on our previous MiniMax-Text-01 model, which contains a total of 456 billion parameters with 45.9 billion parameters activated per token. The M1 model natively supports a context length of 1 million tokens, 8x the context size of DeepSeek R1. Furthermore, the lightning attention mechanism in MiniMax-M1 enables efficient scaling of test-time compute. These properties make M1 particularly suitable for complex tasks that require processing long inputs and thinking extensively. MiniMax-M1 is trained using large-scale reinforcement learning (RL) on diverse problems including sandbox-based, real-world software engineering environments. In addition to M1's inherent efficiency advantage for RL training, we propose CISPO, a novel RL algorithm to further enhance RL efficiency. CISPO clips importance sampling weights rather than token updates, outperforming other competitive RL variants. Combining hybrid-attention and CISPO enables MiniMax-M1's full RL training on 512 H800 GPUs to complete in only three weeks, with a rental cost of just $534,700. We release two versions of MiniMax-M1 models with 40K and 80K thinking budgets respectively, where the 40K model represents an intermediate phase of the 80K training. Experiments on standard benchmarks show that our models are comparable or superior to strong open-weight models such as the original DeepSeek-R1 and Qwen3-235B, with particular strengths in complex software engineering, tool utilization, and long-context tasks. We publicly release MiniMax-M1 at https://github.com/MiniMax-AI/MiniMax-M1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。