轻量高效语言模型，30亿激活参数却媲美大模型性能。
Megrez2 Technical Report
- 跨层共享专家模块，减少参数总量同时保持模型能力。
- 30亿激活参数、75亿存储参数，多项任务表现优于或持平大模型。
- 适合资源受限设备部署，尤其适合移动端和边缘计算场景。
我们提出Megrez2，一种专为设备原生部署优化的轻量级高性能语言模型架构。Megrez2引入跨层专家共享机制，通过复用相邻Transformer层中的专家模块，显著降低总参数量，同时保留大部分模型容量。该架构还采用预门控路由，实现内存高效的专家加载与更快推理速度。作为Megrez2架构的首个实例，Megrez2-Preview在5万亿令牌语料上预训练，并通过监督微调和可验证奖励的强化学习进一步优化。该模型仅需30亿激活参数、75亿存储参数，在语言理解、指令遵循、数学推理和代码生成等多任务上表现竞争力或更优，验证了该架构在准确性、效率与可部署性间的良好平衡，是资源受限场景下实际应用的有力候选。
原文摘要 · Abstract (English)
We present Megrez2, a novel lightweight and high-performance language model architecture optimized for device native deployment. Megrez2 introduces a novel cross-layer expert sharing mechanism, which significantly reduces total parameter count by reusing expert modules across adjacent transformer layers while maintaining most of the model's capacity. It also incorporates pre-gated routing, enabling memory-efficient expert loading and faster inference. As the first instantiation of the Megrez2 architecture, we introduce the Megrez2-Preview model, which is pre-trained on a 5-trillion-token corpus and further enhanced through supervised fine-tuning and reinforcement learning with verifiable rewards. With only 3B activated and 7.5B stored parameters, Megrez2-Preview demonstrates competitive or superior performance compared to larger models on a wide range of tasks, including language understanding, instruction following, mathematical reasoning, and code generation. These results highlight the effectiveness of the Megrez2 architecture to achieve a balance between accuracy, efficiency, and deployability, making it a strong candidate for real-world, resource-constrained applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。