融合多种攻击策略的训练方法,显著提升视觉语言模型抗干扰能力。
A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
- 结合多种攻击方法进行对抗训练,突破传统固定扰动假设
- 微调后CLIP在对抗样本上准确率达43.5%,远超基线的4%
- 适合安全敏感场景如自动驾驶、医疗诊断中的模型加固
视觉语言模型(如CLIP)在自动驾驶、医疗诊断和安防系统等安全关键应用中至关重要,但极易遭受对抗攻击,严重影响其可靠性。现有方法多依赖FGSM、AutoAttack、DeepFool等攻击生成对抗样本,并通过对抗训练提升鲁棒性,但常受限于固定扰动范数或预设攻击模式,且计算成本高。本文提出一种新型对抗训练框架,融合多种攻击策略与先进机器学习技术,显著增强VLM对多样化攻击的防御能力。在真实数据集CIFAR-10和CIFAR-100上的实验表明,微调后的CLIP模型在对抗样本上达到43.5%准确率,而基线模型仅4%;神经网络模型在复杂分类任务中准确率达98%,XGBoost模型预测成功率亦达85.26%。
原文摘要 · Abstract (English)
The robustness of Vision-Language Models (VLMs) such as CLIP is critical for their deployment in safety-critical applications like autonomous driving, healthcare diagnostics, and security systems, where accurate interpretation of visual and textual data is essential. However, these models are highly susceptible to adversarial attacks, which can severely compromise their performance and reliability in real-world scenarios. Previous methods have primarily focused on improving robustness through adversarial training and generating adversarial examples using models like FGSM, AutoAttack, and DeepFool. However, these approaches often rely on strong assumptions, such as fixed perturbation norms or predefined attack patterns, and involve high computational complexity, making them challenging to implement in practical settings. In this paper, we propose a novel adversarial training framework that integrates multiple attack strategies and advanced machine learning techniques to significantly enhance the robustness of VLMs against a broad range of adversarial attacks. Experiments conducted on real-world datasets, including CIFAR-10 and CIFAR-100, demonstrate that the proposed method significantly enhances model robustness. The fine-tuned CLIP model achieved an accuracy of 43.5% on adversarially perturbed images, compared to only 4% for the baseline model. The neural network model achieved a high accuracy of 98% in these challenging classification tasks, while the XGBoost model reached a success rate of 85.26% in prediction tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。