提出细粒度分层混合精度量化方法，提升ViT模型压缩效率
LampQ: Towards Accurate Layer-wise Mixed Precision Quantization for Vision Transformers
- 按层量化+类型感知敏感度评估，实现精准控制
- 在ImageNet等数据集上达到最优压缩性能，精度损失小
- 适合需要高效部署ViT的开发者和研究者
如何准确量化预训练的视觉变换器（Vision Transformer, ViT）？量化算法可将ViT压缩为低比特格式，显著降低内存与计算开销，同时保持较高精度。然而，现有方法多采用统一精度，忽略不同组件对量化敏感度的差异。基于度量的混合精度量化（MPQ）虽具潜力，但先前方法存在三大缺陷：1）粒度粗，2）不同组件间度量尺度不一致，3）比特分配未考虑量化本身。本文提出LampQ（面向视觉变换器的分层混合精度量化），通过逐层量化实现精细控制与高效加速，引入类型感知的基于Fisher信息的度量以衡量敏感度，并利用整数线性规划进行最优比特分配，再迭代优化。大量实验表明，LampQ在图像分类、目标检测及零样本量化等多种任务预训练的ViT中均达到当前最优性能。
原文摘要 · Abstract (English)
How can we accurately quantize a pre-trained Vision Transformer model? Quantization algorithms compress Vision Transformers (ViTs) into low-bit formats, reducing memory and computation demands with minimal accuracy degradation. However, existing methods rely on uniform precision, ignoring the diverse sensitivity of ViT components to quantization. Metric-based Mixed Precision Quantization (MPQ) is a promising alternative, but previous MPQ methods for ViTs suffer from three major limitations: 1) coarse granularity, 2) mismatch in metric scale across component types, and 3) quantization-unaware bit allocation. In this paper, we propose LampQ (Layer-wise Mixed Precision Quantization for Vision Transformers), an accurate metric-based MPQ method for ViTs to overcome these limitations. LampQ performs layer-wise quantization to achieve both fine-grained control and efficient acceleration, incorporating a type-aware Fisher-based metric to measure sensitivity. Then, LampQ assigns bit-widths optimally through integer linear programming and further updates them iteratively. Extensive experiments show that LampQ provides the state-of-the-art performance in quantizing ViTs pre-trained on various tasks such as image classification, object detection, and zero-shot quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。