用进化算法生成更复杂的对抗提示,提升视觉语言模型的鲁棒性
Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models
- 结合梯度法与遗传进化生成多样化对抗提示
- 在多个数据集上优于当前最优对抗提示方法
- 适合关注模型抗攻击能力的研究者
大型预训练视觉语言模型(如CLIP)虽具强大泛化能力,但对对抗样本仍高度敏感。已有工作通过对抗训练优化文本提示,提升鲁棒性与泛化性,但主要依赖单一梯度方向扰动(如PGD),导致对抗样本多样性不足,鲁棒性提升有限。为此,本文提出基于进化的区域对抗提示调优方法ER-APT,融合梯度法与遗传进化机制,生成更丰富、更具挑战性的对抗样本。每轮训练中,先用传统梯度方法生成对抗样本,再通过选择、突变、交叉等遗传操作优化,实现更广域、更激进的扰动分布。最终演化出的对抗样本用于提示调优,实现区域级对抗优化,而非传统单点对抗提示调优。同时提出动态损失权重机制,平衡精度与鲁棒性学习效率。在多个基准数据集上的实验表明,所提方法显著优于现有先进对抗提示方法。代码已公开于https://github.com/jiaxiaojunQAQ/ER-APT。
原文摘要 · Abstract (English)
Large pre-trained vision-language models (VLMs), such as CLIP, demonstrate impressive generalization but remain highly vulnerable to adversarial examples (AEs). Previous work has explored robust text prompts through adversarial training, achieving some improvement in both robustness and generalization. However, they primarily rely on singlegradient direction perturbations (e.g., PGD) to generate AEs, which lack diversity, resulting in limited improvement in adversarial robustness. To address these limitations, we propose an evolution-based region adversarial prompt tuning method called ER-APT, which combines gradient methods with genetic evolution to generate more diverse and challenging AEs. In each training iteration, we first generate AEs using traditional gradient-based methods. Subsequently, a genetic evolution mechanism incorporating selection, mutation, and crossover is applied to optimize the AEs, ensuring a broader and more aggressive perturbation distribution.The final evolved AEs are used for prompt tuning, achieving region-based adversarial optimization instead of conventional single-point adversarial prompt tuning. We also propose a dynamic loss weighting method to adjust prompt learning efficiency for accuracy and robustness. Experimental evaluations on various benchmark datasets demonstrate the superiority of our proposed method, outperforming stateof-the-art APT methods. The code is released at https://github.com/jiaxiaojunQAQ/ER-APT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。