arXiv:2508.02322cs.CLcs.LG2025-08AAAI被引 5

通过细粒度微专家分析,实现MoE模型高效压缩与量化。

CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis

  • 以微专家为单位跨矩阵分析冗余,突破传统专家级压缩瓶颈。
  • 在20%-60%剪枝率下优于主流基线,2比特量化性能超越现有方法。
  • 无需训练、5分钟内完成超大模型(Qwen2-57B)分析,适合部署优化。

具有混合专家(MoE)架构的大语言模型在参数增长时展现出强大的性能扩展能力,但伴随显著的计算与存储开销。值得注意的是,MoE模型的性能提升并未随专家参数增长而线性增强。现有工作尝试通过专家级剪枝、合并或分解来减少参数,但仍面临性能与效率的双重挑战。本文提出将微专家作为更细粒度的压缩单元,跨越矩阵维度进行建模,首次从微观视角将MoE层视为微专家的混合体。我们设计了CAMERA——一个轻量且无需训练的微专家冗余分析框架,发现解码过程中微专家贡献存在显著差异。基于此,进一步提出结构化微专家剪枝框架CAMERA-P和面向微专家的混合精度量化方案CAMERA-Q。在九个下游任务上的实验表明,CAMERA-P在20%至60%剪枝率下持续优于强基线;CAMERA-Q在激进的2比特量化下表现更优,超越现有矩阵与通道级方法。尤其值得一提的是,该方法可在单张NVIDIA A100-40GB GPU上于5分钟内完成对Qwen2-57B-A14B模型的完整微专家分析。

原文摘要 · Abstract (English)

Large Language Models (LLMs) with Mixture-of-Experts (MoE) architectures are distinguished by their strong performance scaling with increasing parameters across a wide range of tasks, yet they also suffer from substantial computational and storage overheads. Notably, the performance gains of MoE models do not scale proportionally with the growth in expert parameters. While prior works attempt to reduce parameters via expert-level pruning, merging, or decomposition, they still suffer from challenges in both performance and computational efficiency. In this paper, we address these challenges by introducing micro-expert as a finer-grained compression unit that spans across matrices. We first establish a more fundamental perspective, viewing MoE layers as mixtures of micro-experts, and present CAMERA, a lightweight and training-free framework for identifying micro-expert redundancy. Our analysis uncovers significant variance in micro-expert contributions during decoding. Based on this insight, we further propose CAMERA-P, a structured micro-expert pruning framework, and CAMERA-Q, a mixed-precision quantization idea designed for micro-experts. Extensive experiments on nine downstream tasks show that CAMERA-P consistently outperforms strong baselines under pruning ratios ranging from 20% to 60%. Furthermore, CAMERA-Q achieves superior results under aggressive 2-bit quantization, surpassing existing matrix- and channel-level ideas. Notably, our method enables complete micro-expert analysis of Qwen2-57B-A14B in less than 5 minutes on a single NVIDIA A100-40GB GPU.

MoE模型模型压缩微专家量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。