用共享防御知识提升大模型抗攻击能力,高效又强。
Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models
- 用超网络生成适配不同防御方法的LoRA,实现知识共享。
- 在多个数据集上显著提升鲁棒性,超越现有方法。
- 适合需要高效增强大视觉模型安全性的研究者。
大型视觉模型易受对抗样本攻击,亟需提升其鲁棒性。尽管对抗训练对深度卷积模型有效,但因计算成本高,在大模型上难以扩展。近期方法如低秩自适应(LoRA)的对抗微调虽有进展,仍无法媲美全参数对抗微调的精度。融合多种防御机制是提升大模型鲁棒性的潜力方向,但尚未充分探索。为此,我们提出超对抗微调(HyperAT),利用不同方法间的共享防御知识,高效且有效地提升模型鲁棒性。具体地,将每种防御方法的对抗微调视为学习任务,由超网络生成对应的LoRA;提出随机采样与微调策略,促进不同防御间的知识迁移;最后合并多样LoRA以增强鲁棒性。在多个数据集和模型架构上的实验表明,HyperAT在不增加过多计算开销的前提下,显著提升了预训练大视觉模型的对抗鲁棒性,建立了新的性能基准。
原文摘要 · Abstract (English)
Large vision models have been found vulnerable to adversarial examples, emphasizing the need for enhancing their adversarial robustness. While adversarial training is an effective defense for deep convolutional models, it often faces scalability issues with large vision models due to high computational costs. Recent approaches propose robust fine-tuning methods, such as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but they still struggle to match the accuracy of full parameter adversarial fine-tuning. The integration of various defense mechanisms offers a promising approach to enhancing the robustness of large vision models, yet this paradigm remains underexplored. To address this, we propose hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously. Specifically, adversarial tuning of each defense method is formulated as a learning task, and a hypernetwork generates LoRA specific to this defense. Then, a random sampling and tuning strategy is proposed to extract and facilitate the defensive knowledge transfer between different defenses. Finally, diverse LoRAs are merged to enhance the adversarial robustness. Experiments on various datasets and model architectures demonstrate that HyperAT significantly enhances the adversarial robustness of pretrained large vision models without excessive computational overhead, establishing a new state-of-the-art benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。