不重训练，几分钟就把大模型变稀疏，提速近1.2倍。
Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
- 分析神经元激活模式，自动拆分密集层为共享与路由专家
- 仅用2000样本微调，计算量不变下提速1.17倍
- 适合部署急迫、资源有限的模型优化场景
扩大语言模型可提升性能，但显著增加推理成本，其中前馈网络（FFN）消耗了大部分计算资源。尽管混合专家（MoE）架构可通过稀疏激活降低成本，但将现有稠密模型重构为MoE通常需在数百亿数据上大规模重训。本文提出一种分析性后训练框架，仅用少量校准数据即可快速将FFN重构为稀疏MoE结构。该方法通过分析神经元激活模式，将神经元划分为始终活跃的共享专家和条件激活的路由专家，并基于代表性神经元统计量解析构建路由器，支持立即部署或轻量微调。该方法既适用于稠密模型，也可递归应用于现有MoE模型以实现层次化稀疏。实验表明，在计算受限场景下可实现最高1.17倍加速，处理时间仅数分钟，微调样本仅2000个，显著优于需数倍资源的方法。
原文摘要 · Abstract (English)
Scaling large language models (LLMs) improves performance but significantly increases inference costs, with feed-forward networks (FFNs) consuming the majority of computational resources. While Mixture-of-Experts (MoE) architectures can reduce this cost through sparse activation, restructuring existing dense models into MoEs typically requires extensive retraining on hundreds of billions of tokens. We propose an analytical post-training framework that rapidly restructures FFNs into sparse MoE architectures using only a small calibration dataset. The method analyzes neuron activation patterns to partition neurons into always-active shared experts and conditionally activated routed experts, then constructs a router analytically from representative neuron statistics, enabling immediate deployment or optional lightweight fine-tuning. This approach applies both to dense models and recursively to existing MoE models for hierarchical sparsity. Experiments demonstrate up to $1.17\times$ speedup in compute-bound scenarios with only minutes of processing and 2k-sample fine-tuning, outperforming methods requiring orders of magnitude more resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。