用图文对齐检测对抗样本，提升图像分类鲁棒性。
Robust image classification with multi-modal large language models
- 利用多模态大模型融合视觉与文本信息判断输入是否一致
- 在CIFAR-10和ImageNet上显著提升防御效果，拒识率更高
- 适合需要高安全性的图像分类场景，如医疗或自动驾驶
深度神经网络易受对抗样本攻击，即精心设计的输入可使模型以高置信度错误分类。为缓解此问题，已有对抗训练和检测防御方法，但多数仅关注单一模态，忽略视觉与文本描述间的关系。本文提出新型防御机制MultiShield，通过多模态大语言模型检测对抗样本，并在视觉与文本表示不一致时拒绝分类。在CIFAR-10和ImageNet数据集上，对鲁棒与非鲁棒模型的大量评估表明，MultiShield可轻松集成，有效检测并拒识对抗样本，优于原始防御方法。
原文摘要 · Abstract (English)
Deep Neural Networks are vulnerable to adversarial examples, i.e., carefully crafted input samples that can cause models to make incorrect predictions with high confidence. To mitigate these vulnerabilities, adversarial training and detection-based defenses have been proposed to strengthen models in advance. However, most of these approaches focus on a single data modality, overlooking the relationships between visual patterns and textual descriptions of the input. In this paper, we propose a novel defense, MultiShield, designed to combine and complement these defenses with multi-modal information to further enhance their robustness. MultiShield leverages multi-modal large language models to detect adversarial examples and abstain from uncertain classifications when there is no alignment between textual and visual representations of the input. Extensive evaluations on CIFAR-10 and ImageNet datasets, using robust and non-robust image classification models, demonstrate that MultiShield can be easily integrated to detect and reject adversarial examples, outperforming the original defenses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。