通过分析令牌空间自相关性,提升ViT模型的鲁棒性和表征能力。
SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers

- 按空间自相关性分组令牌,优化注意力前的特征输入
- 在ImageNet-1K上达到94.9%准确率,多项鲁棒性测试领先
- 无需重训练,直接提升现有ViT模型性能
近年来,视觉变换器(ViTs)在各类视觉识别任务中表现出色。然而,提升其鲁棒性的尝试收效有限,主要依赖训练策略、输入块增强或网络结构改进,往往需要大量训练与微调,耗时且资源密集。为此,本文提出一种新方法——空间自相关令牌分析(SATA)。通过分析令牌特征间的空间自相关性,在自注意力机制的前馈网络(FFN)模块输入前对令牌进行分组,从而增强模型的表征能力和鲁棒性。SATA可无缝集成至现有预训练ViT基线模型,无需重新训练或额外微调,同时降低FFN计算负载。实验表明,引入SATA的基线模型在ImageNet-1K图像分类上达到新的顶尖准确率(top-1=94.9%),并在多个鲁棒性基准测试中创下新纪录:ImageNet-A(top-1=63.6%)、ImageNet-R(top-1=79.2%)、ImageNet-C(mCE=13.6%),均无需额外训练。
原文摘要 · Abstract (English)
Over the past few years, vision transformers (ViTs) have consistently demonstrated remarkable performance across various visual recognition tasks. However, attempts to enhance their robustness have yielded limited success, mainly focusing on different training strategies, input patch augmentation, or network structural enhancements. These approaches often involve extensive training and fine-tuning, which are time-consuming and resource-intensive. To tackle these obstacles, we introduce a novel approach named Spatial Autocorrelation Token Analysis (SATA). By harnessing spatial relationships between token features, SATA enhances both the representational capacity and robustness of ViT models. This is achieved through the analysis and grouping of tokens according to their spatial autocorrelation scores prior to their input into the Feed-Forward Network (FFN) block of the self-attention mechanism. Importantly, SATA seamlessly integrates into existing pre-trained ViT baselines without requiring retraining or additional fine-tuning, while concurrently improving efficiency by reducing the computational load of the FFN units. Experimental results show that the baseline ViTs enhanced with SATA not only achieve a new state-of-the-art top-1 accuracy on ImageNet-1K image classification (94.9%) but also establish new state-of-the-art performance across multiple robustness benchmarks, including ImageNet-A (top-1=63.6%), ImageNet-R (top-1=79.2%), and ImageNet-C (mCE=13.6%), all without requiring additional training or fine-tuning of baseline models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。