加速让视觉语言模型答案不稳,20%的情况会变,6.5%从对变错。
Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study
- 测试8种加速方法在10个数据集上,发现模型回答常变化。
- 加速后20%输入样本答案改变,其中6.5%由正确转错误。
- 医疗模型案例验证不稳定性,适合关注可靠性的开发者看。
视觉语言模型(VLMs)虽强大但计算开销大,为避免重训练,常采用量化、剪枝等后训练加速技术。然而现有评估多关注整体性能小幅下降,忽视关键问题:加速后的模型是否仍对相同输入给出一致答案?这对医疗诊断等需稳定输出的工业场景至关重要。本文系统研究了四种主流模型(LLaVA-1.5、LLaVA-Next、Qwen2-VL、Qwen2.5-VL)在八种加速方法下的表现,覆盖十个多模态基准。结果表明:尽管平均性能下降极小,但加速模型在高达20%的样本中改变了原始答案,其中最多6.5%的变更导致原本正确的答案变为错误。输入扰动加剧了不一致性,医学专用模型LLaVA-Med的案例研究进一步证实该现象。研究揭示加速评估中对实例级稳定性的普遍忽视,亟需建立更严格的可靠性检查机制。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are powerful yet computationally intensive for widespread practical deployments. To address such challenge without costly re-training, post-training acceleration techniques like quantization and token reduction are extensively explored. However, current acceleration evaluations primarily target minimal overall performance degradation, overlooking a crucial question: does the accelerated model still give the same answers to the same questions as it did before acceleration? This is vital for stability-centered industrial applications where consistently correct answers for specific, known situations are paramount, such as in AI-based disease diagnosis. We systematically investigate this for accelerated VLMs, testing four leading models (LLaVA-1.5, LLaVA-Next, Qwen2-VL, Qwen2.5-VL) with eight acceleration methods on ten multi-modal benchmarks. Our findings are stark: despite minimal aggregate performance drops, accelerated models changed original answers up to 20% of the time. Critically, up to 6.5% of these changes converted correct answers to incorrect. Input perturbations magnified these inconsistencies, and the trend is confirmed by case studies with the medical VLM LLaVA-Med. This research reveals a significant oversight in VLM acceleration, stressing an urgent need for instance-level stability checks to ensure trustworthy real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。