用语言模型选攻击目标,让对抗样本更可控、可复现。
Semantically Guided Adversarial Testing of Vision Models Using Language Models
- 用预训练语言模型提取语义相似度,指导对抗攻击的目标选择。
- 在5种攻击方法上,比WordNet等静态资源提升显著,尤其对远距离类别。
- 可生成可解释的对抗测试基准,适合跨模型、跨数据集评估。
在针对视觉模型的定向对抗攻击中,目标标签的选择是决定攻击成败的关键因素,但常被忽视。现有策略多依赖随机性、模型预测或静态语义资源,导致可解释性差、可复现性低或灵活性不足。本文提出一种基于语义引导的对抗目标选择框架,利用预训练语言模型与视觉-语言模型的跨模态知识迁移。我们评估了BERT、TinyLLAMA和CLIP等模型作为相似性来源,选取与真实标签最相关和最不相关的类别,构建最优与最劣对抗场景。在三个视觉模型和五种攻击方法上的实验表明,该框架能持续生成有效的对抗目标,优于WordNet等静态词汇数据库,尤其在远距离类别关系上表现突出。此外,静态目标标签测试可预先评估相似性来源的有效性。结果验证了预训练模型在构建可解释、标准化、可扩展的对抗测试基准方面的适用性,适用于多种模型架构与数据集。
原文摘要 · Abstract (English)
In targeted adversarial attacks on vision models, the selection of the target label is a critical yet often overlooked determinant of attack success. This target label corresponds to the class that the attacker aims to force the model to predict. Now, existing strategies typically rely on randomness, model predictions, or static semantic resources, limiting interpretability, reproducibility, or flexibility. This paper then proposes a semantics-guided framework for adversarial target selection using the cross-modal knowledge transfer from pretrained language and vision-language models. We evaluate several state-of-the-art models (BERT, TinyLLAMA, and CLIP) as similarity sources to select the most and least semantically related labels with respect to the ground truth, forming best- and worst-case adversarial scenarios. Our experiments on three vision models and five attack methods reveal that these models consistently render practical adversarial targets and surpass static lexical databases, such as WordNet, particularly for distant class relationships. We also observe that static testing of target labels offers a preliminary assessment of the effectiveness of similarity sources, \textit{a priori} testing. Our results corroborate the suitability of pretrained models for constructing interpretable, standardized, and scalable adversarial benchmarks across architectures and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。