系统评测多种压缩技术在MoE大模型中的协同效果,揭示组合使用时的隐藏问题。
Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

- 构建端到端压缩流水线,联合评估剪枝、量化与缓存压缩的交互影响。
- 发现单独评估的压缩技术组合后性能大幅下降,且质量损失不可预测。
- 适合部署优化者、模型压缩研究者,尤其关注实际硬件上的效率平衡。
Mixture-of-Experts(MoE)大模型通过稀疏激活高效扩展容量,但其庞大的专家参数量、路由不均衡及长上下文键值缓存增长,使得在普通硬件上部署困难。实际部署常需叠加多种压缩技术:专家剪枝移除冗余专家,权重量化降低内存占用,键值缓存压缩缓解长上下文内存压力。然而,这些技术通常独立评估,缺乏对组合使用时相互作用的研究。本文提出MoEXBench,一个系统性基准测试框架,用于评估可组合的MoE压缩作为端到端部署流程。MoEXBench涵盖10个规模从30B到235B的MoE模型,覆盖标准注意力、混合线性注意力和滑动窗口注意力架构。评估了20%-50%的专家剪枝率、1至16位权重量化方案以及多种键值缓存精度设置,分别及组合应用。引入八模块评估套件,联合衡量可组合压缩质量、工作负载与架构鲁棒性、剪枝/量化/缓存敏感度及在普通硬件上的部署效率。结果表明压缩方法间存在非平凡交互:组合压缩无法由单一技术推断,压缩率不能可靠预测质量损失或运行时间收益,专家剪枝是主要退化来源,平均质量可能掩盖特定工作负载和架构的失败。通过发布归一化模块评分、压缩模型及可复现脚本,MoEXBench支持跨MoE系列与硬件后端的实际精度-内存-延迟对比。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance, and long-context KV-cache growth make deployment difficult on commodity hardware. Practical deployment often requires stacking multiple compression techniques: expert pruning removes redundant experts, weight quantization lowers model memory footprint, and KV-cache compression reduces long-context memory pressure. However, these techniques are typically evaluated in isolation, leaving open how they interact when applied together in realistic deployment pipelines. In this work, we present MoEXBench, a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow. MoEXBench studies 10 MoE models ranging from 30B to 235B total parameters across standard-attention, hybrid linear-attention, and sliding window attention architectures. It evaluates 20%-50% expert pruning rates, 1 to 16 bit weight-quantization schemes, and multiple KV-cache precision settings, applied both individually and in combination. MoEXBench introduces an eight-module evaluation suite that jointly measures composable-compression quality, workload and architecture robustness, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware. Our results reveal non-trivial interactions among compression methods: composable compression cannot be predicted from standalone techniques, compression rate alone does not reliably predict quality loss or runtime gain, expert pruning is the dominant degradation source, and average quality can hide workload and architecture-specific failures. By releasing normalized module scores, compressed artifacts, and reproducible scripts, MoEXBench enables practical accuracy-memory-latency comparison across MoE families and hardware backends.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。