小模型量化部署需考虑硬件协同，别只看参数大小。
Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
- 拆解视觉编码器、投影层、语言模型分别量化，发现结构比规模更影响精度
- MoE架构在INT4下抗噪能力强，密集模型则明显退化；SigLIP在特定硬件上延迟高
- 低精度虽省显存，但解量化开销导致生成变慢，适合资源受限边缘场景
参数少于30亿的视觉语言模型正推动设备端多模态智能落地。然而，对各组件量化特性的深入理解仍是部署瓶颈。本文在Jetson Orin NX和AGX平台上，系统评估了六种量化配置下的五个假设。结果表明：(1) 量化敏感性由结构范式（MoE vs. dense）决定而非规模，MoE架构可缓解INT4噪声，而密集模型性能下降；(2) SigLIP编码器在Jetson Ampere平台上的INT8推理延迟显著升高，是编码器-内核-硬件交互所致，非模型本身缺陷；(3) 虽然LLM的INT4量化大幅降低显存占用，但解量化开销导致生成速度变慢；(4) 多组件量化误差总体近似相加，但跨模态对齐路径误差受架构影响；(5) 不同平台能效比差异大，主要受内存带宽限制。
原文摘要 · Abstract (English)
The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and large language model backbone yields the following results: (1) Quantization sensitivity is governed by the structural paradigm (MoE vs. dense) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; (2) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere--a deployment-specific encoder-kernel-hardware interaction, not a SigLIP flaw; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) Composite quantization errors are largely additive, except along the modality-alignment path, which is architecture-dependent; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。