arXiv:2507.16257cs.CV2025-07中稿 · ed被引 2

用高质量描述文本提升视觉模型抗干扰能力,让模型更鲁棒。

Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models

  • 用高质图文描述生成对抗样本,引导模型学习更全面的视觉特征。
  • 在16个零样本数据集上实现当前最优的鲁棒性与准确率。
  • 适合关注视觉模型安全、语言如何增强视觉理解的研究者。

防御预训练视觉语言模型(如CLIP)免受对抗攻击至关重要,因其广泛应用于图像分类等零样本任务。然而,现有对抗训练方法大多忽视语言对视觉鲁棒性的提升作用:(1) 监督式对抗训练依赖短文本(如类别标签)生成对抗扰动,导致过拟合于训练数据中的物体类别;(2) 无监督对抗训练虽避免过拟合,但因缺乏语义引导,在面对实际文本引导的对抗攻击时表现不佳。为此,我们提出质量文本引导的对抗微调(QT-AFT),在训练中引入高质量图像描述,引导对抗样本避开图像中多样化的语义内容。这使视觉编码器在对抗噪声下仍能稳健识别更广泛的图像特征,从而提升多种下游任务的鲁棒性。QT-AFT克服了先前方法的关键缺陷——监督法的过拟合与无监督法的语义缺失,在16个零样本数据集上实现了最先进的零样本对抗鲁棒性与干净准确率。此外,我们的研究揭示了语言在增强视觉鲁棒性中的关键作用:除物体名称外,描述物体属性可进一步提升零样本鲁棒性。这些发现指明未来方向——将高质量语言监督置于鲁棒视觉表征学习的核心。

原文摘要 · Abstract (English)

Defending pre-trained vision-language models (VLMs), such as CLIP, against adversarial attacks is crucial, as these models are widely used in diverse zero-shot tasks, including image classification. However, existing adversarial training (AT) methods for robust fine-tuning largely overlook the role of language in enhancing visual robustness. Specifically, (1) supervised AT methods rely on short texts (e.g., class labels) to generate adversarial perturbations, leading to overfitting to object classes in the training data, and (2) unsupervised AT avoids this overfitting but remains suboptimal against practical text-guided adversarial attacks due to its lack of semantic guidance. To address these limitations, we propose Quality Text-guided Adversarial Fine-Tuning (QT-AFT), which leverages high-quality captions during training to guide adversarial examples away from diverse semantics present in images. This enables the visual encoder to robustly recognize a broader range of image features even under adversarial noise, thereby enhancing robustness across diverse downstream tasks. QT-AFT overcomes the key weaknesses of prior methods -- overfitting in supervised AT and lack of semantic awareness in unsupervised AT -- achieving state-of-the-art zero-shot adversarial robustness and clean accuracy, evaluated across 16 zero-shot datasets. Furthermore, our comprehensive study uncovers several key insights into the role of language in enhancing vision robustness; for example, describing object properties in addition to object names further enhances zero-shot robustness. Our findings point to an urgent direction for future work -- centering high-quality linguistic supervision in robust visual representation learning.

视觉语言模型对抗鲁棒性语言引导零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。