通过协同压缩技术，让超大规模MoE模型在边缘设备上高效部署。
Collaborative Compression for Large-Scale MoE Deployment on Edge
- 结合专家剪枝、混合精度量化和激活优化实现协同压缩。
- 将DeepSeek-V3模型存储从1.3TB压缩至103GB，保持高质量输出。
- 首次在128GB内存限制下成功部署超大MoE模型，适合边缘计算场景。
Mixture of Experts (MoE) 架构是扩展大型语言模型（LLMs）的重要方法，可在保持低计算成本的同时提升模型容量。然而，超大规模MoE模型仍拥有数千亿参数，需要巨大内存与存储，难以在资源受限的边缘平台部署。仅靠剪枝或量化难以解决此问题，因压缩比过高会导致准确率和输出质量显著下降。为此，我们提出一种协同压缩框架，结合专家剪枝、混合精度量化和激活优化，可将超大规模MoE模型DeepSeek-V3的存储量从1.3TB降至103GB，同时保持高输出质量，且精度优于传统统一低比特量化方法。据我们所知，这是首个在严格128GB总内存限制下成功部署压缩后的DeepSeek-V3模型的工作。我们在多个基准测试和不同内存约束条件下进行的全面实验表明，该方法在更小模型尺寸下实现了更高精度，显著优于均匀低比特量化方法。
原文摘要 · Abstract (English)
The Mixture of Experts (MoE) architecture is an important method for scaling Large Language Models (LLMs). It increases model capacity while keeping computation cost low. However, the ultra-large MoE models still have hundreds of billions of parameters, requiring massive memory/storage and leading to difficulties for deployment on resource-constrained edge platforms. Pruning or quantization alone can hardly address the issue, because of the super-aggressive compression ratio with significantly degraded accuracy and output quality. To facilitate the deployment of ultra-large MoEs on edge platforms, we propose a collaborative compression framework by combining expert pruning, mixed-precision quantization, and activation optimization. It can effectively reduce the storage footprint of the ultra-large MoE DeepSeek-V3 from 1.3TB to 103GB, while preserving high output quality with better accuracy than traditional uniform low-bit quantization methods. To the best of our knowledge, we are the first to deploy a compressed model from the ultra-large DeepSeek-V3 on the platform with a strict 128GB total memory limit. Our comprehensive experiments on multiple benchmarks under various memory constraints demonstrate the effectiveness of our method with smaller model sizes and higher accuracy than uniform low-bit quantization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。