arXiv:2602.09316cs.LG2026-02被引 6

通过分析专家使用频率与信息密度差异,实现更优的MoE模型压缩。

Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density

  • 基于激活频率和有效秩融合度量专家重要性,动态分配压缩资源。
  • 在60%压缩率下,Qwen3-30B模型困惑度降至16.92,优于基线超8.0点。
  • 通过稀疏投影重建残差,以极少参数恢复丢失信息,适合部署优化。

基于混合专家(MoE)的大语言模型虽性能优异,但多专家网络带来的巨大内存开销严重阻碍其实际部署。奇异值分解(SVD)压缩成为有前景的训练后技术,但现有方法多采用均匀秩分配或仅依赖静态权重特性,忽略了MoE模型中专家使用频率与内在信息密度的显著异质性。本文提出RFID-MoE框架,通过利用异质的路由频率与信息密度实现高效压缩。首先引入融合指标,结合专家激活频率与有效秩来衡量专家重要性,在固定预算下自适应提升关键专家组的秩。此外,不丢弃压缩残差,而是通过参数高效的稀疏投影机制进行重建,以极小参数开销恢复损失信息。在多个代表性MoE大模型(如Qwen3、DeepSeekMoE)上,不同压缩比下的大量实验表明,RFID-MoE持续优于当前最优方法(如MoBE、D2-MoE)。尤其在Qwen3-30B模型上,60%压缩率下,PTB数据集困惑度达16.92,相比基线降低超过8.0,零样本HellaSwag准确率提升约8%。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have achieved superior performance, yet the massive memory overhead caused by storing multiple expert networks severely hinders their practical deployment. Singular Value Decomposition (SVD)-based compression has emerged as a promising post-training technique; however, most existing methods apply uniform rank allocation or rely solely on static weight properties. This overlooks the substantial heterogeneity in expert utilization observed in MoE models, where frequent routing patterns and intrinsic information density vary significantly across experts. In this work, we propose RFID-MoE, an effective framework for MoE compression by exploiting heterogeneous Routing Frequency and Information Density. We first introduce a fused metric that combines expert activation frequency with effective rank to measure expert importance, adaptively allocating higher ranks to critical expert groups under a fixed budget. Moreover, instead of discarding compression residuals, we reconstruct them via a parameter-efficient sparse projection mechanism to recover lost information with minimal parameter overhead. Extensive experiments on representative MoE LLMs (e.g., Qwen3, DeepSeekMoE) across multiple compression ratios demonstrate that RFID-MoE consistently outperforms state-of-the-art methods like MoBE and D2-MoE. Notably, RFID-MoE achieves a perplexity of 16.92 on PTB with the Qwen3-30B model at a 60% compression ratio, reducing perplexity by over 8.0 compared to baselines, and improves zero-shot accuracy on HellaSwag by approximately 8%.

MoE压缩SVD压缩专家路由模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。