用扩展嵌入层取代专家层，能更高效提升大模型性能
Scaling Embeddings Outperforms Scaling Experts in Language Models
- 用更大嵌入层替代专家层实现稀疏性扩展
- 新模型参数量685亿，激活30亿，性能超同类MoE模型
- 适合追求推理效率的生成式应用开发
尽管混合专家（MoE）架构已成为大规模语言模型稀疏扩展的标准方案，但其正面临收益递减和系统瓶颈。本文探索将嵌入层扩展作为另一种高效的稀疏扩展维度。通过全面分析与实验，我们识别出嵌入层扩展在某些条件下优于专家扩展的帕累托前沿。系统性地揭示了决定该效果的关键架构因素，包括参数分配、模型宽度与深度的协同作用。结合定制化系统优化与推测解码技术，成功将稀疏性转化为实际推理加速。基于这些发现，我们提出长猫-闪光-轻量版（LongCat-Flash-Lite），一个从头训练的68.5B参数模型，其中约30B参数用于嵌入层。尽管超过30B参数投入嵌入层，该模型不仅超越同参数量级的MoE基线，还在代理任务和代码生成领域表现出色，媲美同类规模现有模型。
原文摘要 · Abstract (English)
While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。