针对视觉Transformer易受对抗攻击问题，提出分层选择性微调方法提升鲁棒性。
SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers
- 只微调最易过拟合的少数层，结合平滑度感知优化
- 清洁与对抗准确率普遍提升5%，最高达20%
- 适用于各类ViT模型，无需改动整体架构
视觉变换器（ViTs）已成为先进计算机视觉与多模态基础模型的核心骨干。尽管性能优异，但ViTs仍对对抗扰动敏感，其脆弱性甚至超过卷积神经网络（CNNs）。此外，庞大的参数量与复杂结构使ViTs极易发生对抗过拟合，损害清洁与对抗准确率。本文提出一种新的分层选择性微调方法SAFER，通过识别并仅对最易过拟合的少数层进行微调，在其余层冻结的前提下，对这些层应用平滑度感知最小化。该方法在多种ViT架构与数据集上均能显著提升性能，典型清洁与对抗准确率提升约5%，部分情况下可达20%。
原文摘要 · Abstract (English)
Vision transformers (ViTs) have become essential backbones in advanced computer vision applications and multi-modal foundation models. Despite their strengths, ViTs remain vulnerable to adversarial perturbations, comparable to or even exceeding the vulnerability of convolutional neural networks (CNNs). Furthermore, the large parameter count and complex architecture of ViTs make them particularly prone to adversarial overfitting, often compromising both clean and adversarial accuracy. This paper mitigates adversarial overfitting in ViTs through a novel, layer-selective fine-tuning approach: SAFER. Instead of optimizing the entire model, we identify and selectively fine-tune a small subset of layers most susceptible to overfitting, applying sharpness-aware minimization to these layers while freezing the rest of the model. Our method consistently enhances both clean and adversarial accuracy over baseline approaches. Typical improvements are around 5%, with some cases achieving gains as high as 20% across various ViT architectures and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。