将大模型转为专家混合架构，动态控制激活参数量。
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
- 通过可微分动态剪枝，将密集模型转为专家混合结构。
- 无需微调即在多个模型上超越已有剪枝方法性能。
- 适合资源受限场景下高效部署大模型的开发者。
大语言模型在处理复杂任务方面表现出色，但其巨大的计算和内存开销给在资源受限设备上部署或高效服务带来了挑战。以往方法通过永久移除不重要结构来缓解问题，但常因参数永久删除导致性能显著下降。本文提出一种可微分动态剪枝方法，使密集模型在不永久删除参数的前提下，通过将MLP层转换为专家混合（MoE）架构，始终保持固定数量的活跃参数。该方法无需微调，在Phi-2、LLaMA-2、LLaMA-3和Qwen-2.5等多种模型上均持续优于现有结构剪枝技术。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。