针对视觉语言模型低比特量化中的模态差异问题，提出通道解耦新方法。
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
- 按模态分离异常通道，减少跨模态分布不均影响
- 在W3A3下保持93.5%浮点精度（74.3→69.5）
- 适用于资源受限设备部署高级多模态模型
低比特后训练量化（PTQ）是将视觉语言模型（VLMs）部署到资源受限设备的关键技术。然而，现有方法因文本与视觉模态激活分布异质性导致性能下降。我们发现这种跨模态异质性在通道间分布不均：少数通道包含多数模态特异性异常值，且各模态的异常通道通常不同。为此，提出SplitQ框架，核心为模态特异性异常通道解耦（MOCD）模块，能以极小开销有效隔离关键异常通道。为进一步缓解剩余跨模态分布差异，设计自适应跨模态校准（ACC）模块，采用双轻量可学习分支动态修正模态引发的量化误差。在6个主流多模态数据集上大量实验表明，SplitQ在所有量化设置（包括W4A8、W4A4、W3A3、W3A2）下显著优于现有方法。尤其在挑战性的W3A3设置下，仅损失6.5%精度（69.5 vs. 74.3），保持93.5%浮点性能，推动了先进VLM高效部署的边界。代码已开源。
原文摘要 · Abstract (English)
Low-bit post-training quantization (PTQ) is a pivotal technique for deploying Vision-Language Models (VLMs) on resource-constrained devices. However, existing PTQ methods often degrade VLMs' accuracy due to the heterogeneous activation distributions of text and vision modalities during quantization. We find that this cross-modal heterogeneity is distributed unevenly across channels: a small subset of channels contains most modality-specific outliers, and these outliers typically reside in different channels for each modality. Motivated by this, we propose SplitQ, a channel-Splitting-driven post-training Quantization framework. At its core, SplitQ introduces a novel Modality-specific Outlier Channel Decoupling (MOCD) module that effectively isolates salient modality-specific outlier channels with minimal overhead. To further address the remaining cross-modal distribution discrepancies, we design an Adaptive Cross-Modal Calibration (ACC) module that employs dual lightweight learnable branches to dynamically mitigate modality-induced quantization errors. Extensive experiments on popular VLMs demonstrate that SplitQ significantly outperforms existing approaches across 6 popular multi-modal datasets under all evaluated quantization settings, including W4A8, W4A4, W3A3, and W3A2. Notably, SplitQ preserves 93.5% of FP16 performance under the challenging W3A3 setting (69.5 vs. 74.3), pushing the efficiency frontier for deploying advanced VLMs. Our code is available at https://github.com/EMVision-NK/SplitQ
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。