通过多视角归一化与混合令牌增强视觉Transformer效率与性能。
MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformers
- 引入多视角归一化（MVN）融合三种归一化特征，提升特征多样性。
- 设计多尺度卷积令牌混合器（MVTM），在各阶段自适应感受野，高效捕捉视觉模式。
- 在相同或更少参数下，图像分类、检测与分割任务均超越现有SOTA模型。
当前研究致力于提升视觉Transformer（ViTs）的效率。多数工作仅关注有效的令牌混合机制，忽略了其与归一化之间的潜在关联。为增强特征学习的多样性，本文提出两个组件：多视角归一化（MVN）和多视角令牌混合器（MVTM）。MVN通过可学习加权求和融合批归一化、层归一化与实例归一化的特征输出，生成不同分布的特征，提供多样模式信息以促进令牌混合器的协同效应。MVTM采用基于卷积的多尺度混合结构，包含局部、中程与全局滤波器，并在不同阶段配置差异化感受野，有效捕获视觉模式的多层次范围。本文提出新型视觉Transformer模型——多视觉变换器（MVFormer），将其应用于元变换器块（MetaFormer block）这一通用框架。实验表明，相较于当前最先进的基于卷积的ViTs，MVFormer在图像分类、目标检测、实例与语义分割任务上表现更优，且参数量与乘加操作数（MACs）相当或更低。特别地，其变体MVFormer-T、S、B在ImageNet-1K基准上分别达到83.4%、84.3%、84.6%的top-1准确率。
原文摘要 · Abstract (English)
Active research is currently underway to enhance the efficiency of vision transformers (ViTs). Most studies have focused solely on effective token mixers, overlooking the potential relationship with normalization. To boost diverse feature learning, we propose two components: a normalization module called multi-view normalization (MVN) and a token mixer called multi-view token mixer (MVTM). The MVN integrates three differently normalized features via batch, layer, and instance normalization using a learnable weighted sum. Each normalization method outputs a different distribution, generating distinct features. Thus, the MVN is expected to offer diverse pattern information to the token mixer, resulting in beneficial synergy. The MVTM is a convolution-based multiscale token mixer with local, intermediate, and global filters, and it incorporates stage specificity by configuring various receptive fields for the token mixer at each stage, efficiently capturing ranges of visual patterns. We propose a novel ViT model, multi-vision transformer (MVFormer), adopting the MVN and MVTM in the MetaFormer block, the generalized ViT scheme. Our MVFormer outperforms state-of-the-art convolution-based ViTs on image classification, object detection, and instance and semantic segmentation with the same or lower parameters and MACs. Particularly, MVFormer variants, MVFormer-T, S, and B achieve 83.4%, 84.3%, and 84.6% top-1 accuracy, respectively, on ImageNet-1K benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。