对比五种注意力机制，发现双随机注意力在噪声数据下最稳定。
Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations
- 用五种注意力机制对比实验，测试其抗噪声能力。
- 双随机注意力在污染数据上表现最优，最高领先5.1%。
- 适合处理标注不全或含干扰的视觉任务场景。
自注意力机制是Transformer架构的核心，支撑其在各类任务中的成功。尽管存在多种自注意力变体，但它们对噪声和虚假相关性的鲁棒性尚未得到充分研究。本研究在视觉Transformer中评估了Softmax、Sigmoid、Linear、Doubly Stochastic和Cosine注意力机制，在CIFAR-10、CIFAR-100和Imagenette数据集上的不同数据损坏场景下进行测试。结果表明，双随机注意力最为鲁棒：在训练数据或训练与测试数据同时被污染时，其性能始终优于次优机制0.1%至5.1%。该发现为在不完美数据环境下选择自注意力机制提供了依据。代码已公开于https://github.com/ctamayor/NeurIPS-Robustness-ViT。
原文摘要 · Abstract (English)
Self-attention mechanisms are foundational to Transformer architectures, supporting their impressive success in a wide range of tasks. While there are many self-attention variants, their robustness to noise and spurious correlations has not been well studied. This study evaluates Softmax, Sigmoid, Linear, Doubly Stochastic, and Cosine attention within Vision Transformers under different data corruption scenarios. Through testing across the CIFAR-10, CIFAR-100, and Imagenette datasets, we show that Doubly Stochastic attention is the most robust. It consistently outperformed the next best mechanism by $0.1\%-5.1\%$ when training data, or both training and testing data, were corrupted. Our findings inform self-attention selection in contexts with imperfect data. The code used is available at https://github.com/ctamayor/NeurIPS-Robustness-ViT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。