针对多模态专家模型压缩,提出按模态分解的量化方法,显著降低显存占用且性能损失小。
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs

- 按视觉与文本模态分解专家选择频率,消除冗余视觉数据干扰
- 在W3A16下平均性能损失仅2.9%,2比特设置下效果更优
- 适合需高效部署多模态专家模型的研究者与工程师
混合专家多模态大语言模型(MoE-MLLMs)虽表现优异,但带来高昂的GPU显存开销,压缩至关重要。现有基于预测后量化(PTQ)的专家级混合精度量化方法在MoE-MLLMs上效果不佳,源于对专家重要性估计的两个被忽视偏差:(1) 跨模态层面,视觉标记数值主导导致专家选择频率被视觉占优,掩盖对文本关键的专家;(2) 视觉内部层面,大量冗余视觉标记进一步扭曲频率统计,遮蔽对有效视觉内容关键的专家。为此,我们提出MODE——一种面向MoE-MLLMs的模态分解专家级混合精度量化框架。该方法按模态分解专家选择频率,过滤冗余视觉标记以获得去噪后的视觉频率,并结合各模态量化敏感度作为补充信号。这些信号整合进整数线性规划模型,在给定预算下为每个专家分配比特位。大量实验表明,MODE特别适用于MoE-MLLMs,W3A16下平均性能损失控制在2.9%以内,极端2比特设置下收益更大。
原文摘要 · Abstract (English)
Mixture-of-Experts Multimodal Large Language Models (MoE-MLLMs) offer remarkable performance but incur prohibitive GPU memory costs, making compression essential. Among PTQ methods, expert-level mixed-precision quantization has proven effective for MoE-LLMs, yet suffers notable degradation on MoE-MLLMs due to two overlooked biases in expert importance estimation. (1) At the cross-modal level, the numerical dominance of vision tokens causes expert selection frequency to be dominated by vision tokens, masking experts that are critical to the text modality; (2) at the intra-vision level, the large proportion of redundant vision tokens further skew frequency statistics, obscuring experts critical for informative visual content. To bridge gaps, we propose MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE-MLLMs that decomposes expert selection frequency by modality, filters redundant vision tokens to obtain denoised visual frequency, and further evaluates quantization sensitivity per modality as a complementary signal to frequency-based estimation. These signals are integrated into an Integer Linear Programming formulation to assign per-expert bit-widths under a given budget. Extensive experiments show that MODE is particularly well-suited for MoE-MLLMs, limiting average performance loss to within 2.9% at W3A16, with larger gains at the extreme 2-bit setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。