让视觉Transformer自动保持对称性，提升性能和数据效率。
Vanilla Group Equivariant Vision Transformer: Simple and Effective
- 系统改造ViT关键模块，实现整体等变性
- 在多种视觉任务中显著提升性能与数据效率
- 可直接替换现有模型，适合追求鲁棒性的研究者
将对称性先验作为归纳偏置引入视觉Transformer设计，是提升性能的有前景方向。然而，现有等变ViT难以在性能与等变性间取得平衡，主要源于在ViT多样化模块中实现整体等变性困难，尤其在自注意力机制与图像块嵌入的协同方面。为此，我们提出一种简洁框架，系统性地使关键ViT组件——包括图像块嵌入、自注意力、位置编码及下采样/上采样模块——具备等变性，从而构建出保证等变性的视觉Transformer。该架构为即插即用式替换方案，理论严谨且实用性强，甚至可无缝扩展至Swin Transformer。大量实验表明，我们的等变ViT在广泛视觉任务中持续提升性能与数据效率。
原文摘要 · Abstract (English)
Incorporating symmetry priors as inductive biases to design equivariant Vision Transformers (ViTs) has emerged as a promising avenue for enhancing their performance. However, existing equivariant ViTs often struggle to balance performance with equivariance, primarily due to the challenge of achieving holistic equivariant modifications across the diverse modules in ViTs-particularly in harmonizing the Self-Attention mechanism with Patch Embedding. To address this, we propose a straightforward framework that systematically renders key ViT components, including patch embedding, self-attention, positional encodings, and Down/Up-Sampling, equivariant, thereby constructing ViTs with guaranteed equivariance. The resulting architecture serves as a plug-and-play replacement that is both theoretically grounded and practically versatile, scaling seamlessly even to Swin Transformers. Extensive experiments demonstrate that our equivariant ViTs consistently improve performance and data efficiency across a wide spectrum of vision tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。