将密集MLP转为静态专家混合，零训练实现高效推理
MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
- 用张量切片重构密集MLP为静态专家混合结构
- 80亿模型删20%参数，困惑度仅上升2%以内
- 无需训练/校准数据，直接作用于现有模型权重
大型语言模型普遍采用密集变换器架构，每个前馈块的所有参数在每令牌处理中均被激活，虽结构简单但计算效率低。本文提出MLPMoE（MLP专家混合），一种无需训练的确定性转换方法，将变压器块中的密集MLP重构为静态高基数专家混合。该方法基于简单的张量切片与求和操作，将张量并行的代数结构重新诠释为拓扑转换而非分布式训练模式。引入分形消退（差分分支稀疏）和补偿修剪（方差保持分支缩减）作为轻量级结构化稀疏机制。在Qwen2.5-0.5B-Instruct和DeepSeek-R1-Distill-Llama-8B上，零样本转换对代理困惑度影响小于0.05%，参数总量基本不变；在8B模型上，差分稀疏移除约20%的MLP参数，同时困惑度维持在密集基线的2%以内。该方法完全后置运行，不依赖梯度、校准集或路由器训练。代码已公开。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token. While architecturally simple, this is computationally inefficient, since inference costs scale linearly with parameter count. Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks, but these approaches typically rely on clustering, activation profiling, singular value decomposition, or custom routing that requires calibration data. This paper introduces MLPMoE (MLP Mixture-of-Experts), a training-free, deterministic transformation that restructures the dense MLP in transformer blocks into a static, high-cardinality mixture of experts. The transformation uses simple tensor slicing and summation, reinterpreting the algebra of tensor parallelism as a topological conversion rather than a distributed training pattern. We further introduce Fractal Fade (differential branch sparsity) and Compensated Pruning (variance-preserving branch reduction) as lightweight mechanisms for structured sparsity. On Qwen2.5-0.5B-Instruct and DeepSeek-R1-Distill-Llama-8B, the zero-shot MLPMoE transform changes a proxy perplexity metric by less than 0.05 percent while keeping the parameter count effectively constant. On the 8B model, differential sparsity removes about 20 percent of MLP parameters while keeping perplexity within about 2 percent of the dense baseline. The method operates entirely post hoc on existing checkpoints and does not require gradients, calibration sets, or router training. Code is available at https://gist.github.com/iwallarm/fc2ef1eddf226ca7814f9e5e2ae9bad1
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。