通过添加前缀令牌解决视觉编码器量化中的异常值问题。
Activation Quantization of Vision Encoders Needs Prefixing Registers
- 在视觉编码器中插入无意义前缀令牌，抑制其他令牌的异常值。
- 4比特量化下性能显著提升，跨多种模型保持稳定增益。
- 无需训练，可直接集成到现有量化方法中，适合部署优化场景。
大型预训练视觉编码器是多模态智能的核心，支撑从设备端视觉处理到视觉语言模型的应用。由于这些应用常需实时处理海量视觉数据，降低视觉编码器的推理开销至关重要。量化提供了一种实用路径，但即使在8比特精度下仍面临所谓异常值的挑战。本文提出无需训练的RegCache算法，通过在视觉编码器中引入语义无关的前缀令牌，防止其他令牌产生异常值。我们发现视觉编码器的异常值行为与语言模型不同，由此提出中层前缀注入和令牌删除两项技术改进。实验表明，该方法在多种视觉编码器上均能显著提升量化模型性能，尤其在极低比特（如4比特）场景下效果突出。
原文摘要 · Abstract (English)
Large pretrained vision encoders are central to multimodal intelligence, powering applications from on-device vision processing to vision-language models. Since these applications often demand real-time processing of massive visual data, reducing the inference cost of vision encoders is critical. Quantization offers a practical path, but it remains challenging even at 8-bit precision due to so-called outliers. In this work, we propose $\textit{RegCache}$, a training-free algorithm that mitigates outliers in large-scale pretrained vision encoders and serves as a plug-in module that can be applied on top of other quantization methods. RegCache introduces outlier-prone yet semantically meaningless prefix tokens to the vision encoder, which prevent other tokens from having outliers. Notably, we observe that outliers in vision encoders behave differently from those in language models, motivating two technical innovations: middle-layer prefixing and token deletion. Experimental results show that our method consistently improves quantized model performance across various vision encoders, particularly in extremely low-bit regimes (e.g., 4-bit).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。