通过无关语义替换文本，防御视觉语言模型的字体攻击。
QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing
- 识别可能干扰查询的文本区域并用无关语义内容替换
- 在四个模型上实现67.9%-75.0%的恢复率，误伤率仅0.5%-1.1%
- 无需训练、不依赖模型内部结构，适用于闭源模型
字体攻击通过向图像注入误导性文字，使视觉语言模型（VLMs）依赖对抗性文本线索而非视觉证据，构成严重威胁。现有防御方法通常需模型定制修改、额外训练或访问内部组件，难以应用于现代闭源VLM。本文提出QuISE，一种模型无关、无需训练的黑盒防御机制，基于查询无关的语义编辑。QuISE首先通过影响感知文本定位识别可能影响当前查询的文本区域；随后将这些区域替换为与查询和图像均无关的两个语义不同的替代文本；最终通过编辑后图像的答案一致性决定输出。在三个字体攻击基准、四种攻击设置及四类VLM上的实验表明，QuISE能持续提升防御准确率，实现67.9%-75.0%的恢复率，同时保持0.5%-1.1%的误伤率。
原文摘要 · Abstract (English)
Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。