arXiv:2608.07814cs.LGcs.AI2026-08

通过分组专家保留核心权重,实现无重训练的高效模型压缩。

Shape Mutating Expert Compression:LorExperts and BTExperts

论文配图:Shape Mutating Expert Compression:LorExperts and BTExperts
图 1 · 摘自论文原文
  • 将专家按功能聚类,仅保留每组主专家,其余用低秩修正表示。
  • 在50%专家压缩率下,下游任务准确率优于现有方法,专家越多优势越明显。
  • 无需重训路由器,适合部署在资源受限场景的MoE模型优化。

Mixture-of-Experts(MoE)语言模型在保持低每词计算量的同时实现高容量,但部署成本高需压缩其大量专家权重矩阵。专家剪枝(如REAP)和合并会牺牲精度并需重训路由器;低秩增量分解(如D²-MoE)虽保留所有专家与路由器,但随着专家数量增加,性能急剧下降,因单一共享组件难以拟合多个近正交专家。我们发现,尽管专家权重近正交,它们仍形成与权重相似性无关的功能协同激活群组。基于此,提出LorExperts:对专家聚类,每组保留一个全精度主导专家,其余以低秩修正表示,且不改变原有路由器(无需重训)。在Qwen3-30B-A3B和Gemma-4-26B-A4B上,约50%专家压缩率下,该方法在多数任务中保持更优的下游精度与困惑度,且相比D²-MoE的性能差距随专家数E增长而扩大。进一步提出重建微调流程及BTExperts——一种主导与修正的树状结构,支持推理时共享计算开销摊销。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices. Expert pruning (e.g., REAP) and merging reduce cost but sacrifice accuracy and require retraining the router; low-rank delta decomposition of experts (e.g., D^2-MoE) preserves all experts and the router, but degrades sharply as the expert count grows because a single shared component cannot approximate many near-orthogonal experts. Because MoE expert weights are near-orthogonal, a single shared component (as in prior delta decomposition) scales poorly with the expert count; we show that experts nonetheless organize into functional co-activation communities that are decoupled from weight similarity. Building on this, we introduce LorExperts, a router-preserving compression method that clusters experts, keeps one full-precision dominant per cluster, and represents the remaining members as low-rank corrections to their local dominant. LorExperts retains all experts and the original router (no router retraining). At ~50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves downstream accuracy and perplexity better than the baselines on most of the tasks; the margin over D^2-MoE grows with expert count E. We further give a reconstruction fine-tuning procedure for LorExperts, and BTExperts, a tree organization of dominants and corrections that enables inference-time amortization of shared computation.

MoE压缩专家聚类低秩修正无重训

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。