腾讯开源520亿激活参数的超大MoE模型，性能逼近4050亿参数模型。
Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent
- 采用混合专家路由与专家专属学习率策略，提升参数效率。
- 在多任务基准测试中超越700亿参数模型，接近4050亿参数模型表现。
- 适合研究大规模MoE模型与长文本处理的开发者和研究人员。
本文介绍Hunyuan-Large，目前最大的开源基于Transformer的混合专家（MoE）模型，总参数量3890亿，激活参数达520亿，支持最高256K tokens输入。我们在语言理解与生成、逻辑推理、数学求解、编程、长文本处理及综合任务等多个基准上全面评估其性能，结果表明其优于LLama3.1-70B，并在多数任务上与更大规模的LLama3.1-405B模型表现相当。关键实践包括远超以往文献规模的合成数据训练、混合专家路由策略、键值缓存压缩技术以及专家级学习率策略。此外，我们还研究了MoE模型的缩放规律与学习率调度，为未来模型开发提供重要参考。代码与模型权重已公开，以促进后续创新与应用。
原文摘要 · Abstract (English)
In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。