让视觉Transformer里的分类标记更专注,提升分割精度。
Revisiting [CLS] and Patch Token Interaction in Vision Transformers
- 为分类标记和图像块标记设计不同处理路径,避免计算混淆。
- 分割任务性能提升超2个mIoU点,分类准确率不变。
- 仅增加8%参数,无额外计算开销,适配多种模型规模。
视觉Transformer作为强大的可扩展表征学习器,通常在输入序列前添加可学习的[CLS]分类标记以捕捉全局与局部特征。尽管二者性质不同,但模型中对它们的处理方式相同。本文通过分析不同预训练策略下分类标记与图像块标记的交互,发现标准归一化层隐式区分了两类标记。基于此,我们提出针对性的处理路径,专门解耦归一化层及早期查询-键-值投影中的计算流程。该方法显著提升了图像块表征质量,密集预测任务中分割性能提升超过2个mIoU点,同时保持分类精度。实验表明,修改仅引入8%参数增量,无额外计算开销。通过全面消融分析,揭示了哪些组件最受益于这种专化,并验证了方法在不同模型尺度和学习框架下的泛化能力。
原文摘要 · Abstract (English)
Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model. In this work, we investigate the friction between global and local feature learning under different pre-training strategies by analyzing the interactions between class and patch tokens. Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, we propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early query-key-value projections. This targeted specialization leads to significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains of over 2 mIoU points on standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8% increase in parameters, with no additional computational overhead. Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how our approach generalizes across model scales and learning frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。