Mixtral 8x7B用专家混合模型实现高效大模型，性能超越多个主流模型。
Mixtral of Experts
- 每层有8个专家，每个输入仅激活2个专家，动态选择提升效率。
- 推理时仅使用130亿活跃参数，但总参数达470亿，上下文长度支持32k tokens。
- 指令微调版在数学、编程、多语言任务上均超越GPT-3.5等主流模型。
我们提出 Mixtral 8x7B，一种稀疏专家混合（SMoE）语言模型。其架构与 Mistral 7B 相同，每层包含8个前馈块（即专家）。每个令牌在每层由路由网络选择两个专家处理，并融合输出。尽管每个令牌仅激活两个专家，但不同时间步可选择不同专家，因此每个令牌可访问470亿参数，但推理时仅使用130亿活跃参数。模型训练上下文长度为32,000个令牌，在所有评估基准中表现优于或匹配 Llama 2 70B 和 GPT-3.5，尤其在数学、代码生成和多语言任务上显著超越 Llama 2 70B。我们还提供了指令微调版本 Mixtral 8x7B - Instruct，该模型在人类评测中超越 GPT-3.5 Turbo、Claude-2.1、Gemini Pro 和 Llama 2 70B - chat 模型。基础版与指令版均采用 Apache 2.0 许可证开源。
原文摘要 · Abstract (English)
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。