动态路由稀疏专家模型，让大模型更省计算资源且不丢知识。
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
- 将前馈层分块，按输入复杂度动态选择专家。
- 同等算力下比剪枝和传统MoE在生成任务上表现更好。
- 揭示分层激活规律，为未来专家结构设计提供新思路。
随着大语言模型持续扩展，计算成本与资源消耗已成为重大挑战。现有稀疏化方法如剪枝虽能降低计算开销，但可能因参数移除导致模型知识丢失。本文提出DSMoE（动态稀疏专家混合模型），通过将预训练的前馈网络层分割为计算模块实现稀疏化。采用Sigmoid激活与直通估计器实现自适应专家路由，使不同输入可灵活调用模型知识的不同方面。同时引入稀疏性损失项以平衡性能与效率。在LLaMA系列模型上的大量实验表明，在相同计算约束下，DSMoE在语言建模与下游任务中均优于现有剪枝与MoE方法，尤其在生成任务中表现突出。分析显示，DSMoE学习到独特的分层激活模式，为未来MoE架构设计提供了新见解。
原文摘要 · Abstract (English)
As large language models continue to scale, computational costs and resource consumption have emerged as significant challenges. While existing sparsification methods like pruning reduce computational overhead, they risk losing model knowledge through parameter removal. This paper proposes DSMoE (Dynamic Sparse Mixture-of-Experts), a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge based on input complexity. Additionally, we introduce a sparsity loss term to balance performance and computational efficiency. Extensive experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches across language modeling and downstream tasks, particularly excelling in generation tasks. Analysis reveals that DSMoE learns distinctive layerwise activation patterns, providing new insights for future MoE architecture design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。