压缩版视觉语言模型失效模式更独特，尤其在图像模糊时表现异常。
Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption
- 用三类错误分类诊断压缩模型故障，发现语义漂移最常见
- 小模型在模糊图像上错误率高出12.5个百分点，且对否定句反应极端
- 适合关注边缘部署安全性的研究人员和工程师
为边缘部署压缩大型视觉语言模型（VLM）带来了新问题：紧凑模型是否以不同方式失败，而不仅仅是更频繁出错？本研究对比了70亿参数量化模型（Qwen2.5-VL-7B，4-bit NF4）与5亿参数FP16模型（SmolVLM2-500M），在VQAv2和COCO Captions共4000个样本上进行评估。采用三类错误分类（物体盲视、语义漂移、先验偏差）作为诊断框架。通过文本仅依赖的GPT-4o判断，发现语义漂移在VQAv2及Qwen上占主导；在COCO上，SmolVLM2呈现物体盲视与语义漂移混合模式；先验偏差仅在VQAv2中出现。使用几何平均词概率计算预期校准误差（ECE），通过四种结构化否定模板探测组合推理能力，并设计模糊鲁棒性实验。结果显示，紧凑模型表现出显著不同的失效特征：否定崩溃程度高出12.5个百分点（-33.2pp vs. -20.8pp，Wald 95% CI [8.2, 16.8]pp，p < 10^-8），主要由COCO数据驱动，而VQAv2差异不显著（4.5pp，p=0.19）。最具区分力的模板为false_yn：SmolVLM2-500M在100%的COCO测试中错误回答“是”（声称不存在的物体），而Qwen仅14%。分析还揭示数据集相关的非对称校准误差及两次可控消融的模糊实验结果。完整可复现流程已公开，供压缩VLM边缘部署前系统性安全审计。
原文摘要 · Abstract (English)
The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions. A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework. A text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability, compositional reasoning is probed with structured negation probes across four templates, and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse (-33.2pp vs. -20.8pp, Wald 95% CI [8.2, 16.8]pp, p < 10^-8), driven almost entirely by COCO while the VQAv2 gap is not statistically significant (4.5pp, p=0.19). The most discriminating template is false_yn: SMOLVLM2-500M responds "Yes" (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Q WEN 2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。