为视觉语言模型专家架构设计混合精度量化，精准分配每位专家比特数。
MoPEQ: Mixture of Mixed Precision Quantized Experts
- 基于海森迹近似分析专家敏感度，动态分配不同比特精度。
- 在多模型测试中实现精度相近但内存减少30%以上。
- 适合资源受限场景下部署大型视觉语言模型的工程师参考。
采用专家混合（MoE）架构的大规模语言与视觉模型在部署时面临显著的计算和内存挑战。混合精度量化根据模型各层的敏感性和重要性，为不同层分配不同精度。本文提出一种后训练量化算法MoPEQ，为每个专家分配最优位宽。该方法通过海森迹近似分析每个专家的敏感度，而非依赖专家激活频率，从而在保持性能的同时降低内存开销。该逐专家粒度的方法将相似专家聚类，有效压缩模型。在VLMEvalKit基准数据集上，使用最先进模型Deepseek-VL2-tiny、-small、-base及MolmoE进行实验，结果表明，混合精度量化后的MoE模型在精度与均匀精度基线相当的前提下，内存占用大幅降低。我们进一步研究了专家激活频率与敏感度的影响，分别在层级与模型级对2、3、4比特精度分配进行了全面分析，深入理解了视觉语言模型-专家混合架构的混合精度量化机制。
原文摘要 · Abstract (English)
Large Language and Vision Models using a Mixture-of-Experts (MoE) architecture pose significant challenges for deployment due to their computational and memory demands. Mixed Precision Quantization assigns different precisions to different layers of an LLM/VLM based on layer sensitivity and importance within the model. In this work, we propose a Post Training Quantization algorithm, MoPEQ, that assigns optimal bit width to each expert. Our method balances accuracy and model size by analyzing each expert's sensitivity using Hessian trace approximation instead of relying on the activation frequency of the expert. This per-expert granularity approach clusters similar experts to maintain model performance while reducing memory requirements. The experimental results on VLMEvalKit benchmark datasets using State-of-the-art VLMs Deepseek-VL2 -tiny, -small, -base, and MolmoE models demonstrate that our mixed precision quantized MoEs achieve competitive accuracy with substantial improvements in memory footprint compared to uniform-precision baseline methods. We perform a comprehensive study to analyze the impact of expert activation frequency and sensitivity using Hessian trace approximation at both layer-wise and model-wide expert precision allocation of 2, 3, and 4 bits to provide a thorough understanding of mixed precision quantization of VLM-MoEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。