用两阶段检测法高效识别视觉攻击,不训练也能防住误导性图像。
VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense
- 先用无损图像变换快速筛除大部分正常输入
- 对难判案例通过文本嵌入差异检测,准确率达98.7%
- 仅在必要时调用大模型,整体效率高适合部署
大型视觉-语言模型(LVLMs)易受细微扰动的对抗图像影响,导致输出看似合理却错误。本文提出一种通用、高效且无需训练的防御方法,结合图像变换与代理式数据融合,恢复模型正确行为。核心是两阶段检测机制:第一阶段以极低计算成本评估内容保持变换下的图像一致性;第二阶段针对复杂情况,在文本嵌入空间中分析差异。仅当必要时才调用强大大模型处理攻击引发的分歧。关键思想是融合多个响应,利用其共性与差异。实验表明,该方法在保持显著效率的同时达到当前最优检测精度:多数正常图像跳过昂贵处理,即便存在大量对抗样本,额外开销仍极小。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) can be vulnerable to adversarial images that subtly bias their outputs toward plausible yet incorrect responses. We introduce a general, efficient, and training-free defense that combines image transformations with agentic data consolidation to recover correct model behavior. A key component of our approach is a two-stage detection mechanism that quickly filters out the majority of clean inputs. We first assess image consistency under content-preserving transformations at negligible computational cost. For more challenging cases, we examine discrepancies in a text-embedding space. Only when necessary do we invoke a powerful LLM to resolve attack-induced divergences. A key idea is to consolidate multiple responses, leveraging both their similarities and their differences. We show that our method achieves state-of-the-art accuracy while maintaining notable efficiency: most clean images skip costly processing, and even in the presence of numerous adversarial examples, the overhead remains minimal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。