通过引入摘要令牌打破视觉变换器的排列不变性，探索无需位置编码的性能可能。
ViT Registers and Fractal ViT
- 用注意力掩码连接常规令牌与摘要令牌，破坏输入顺序的不变性。
- 在图像分类任务中未超越带寄存器的ViT，性能提升不显著。
- 适用于研究位置编码作用机制或探索轻量化视觉模型的场景。
受近期研究启发，包括语言模型中无位置编码（NoPE）仍表现出色，以及额外的丢弃令牌（寄存器）可提升大型视觉变换器（ViT）性能，我们提出并测试了一种名为分形ViT的新变体。该模型通过在常规令牌与‘摘要令牌’之间应用注意力掩码，打破令牌间的排列不变性，单独使用或结合多种位置编码方式。实验结果表明，这些模型并未优于带寄存器的ViT，说明相关发现可能具有规模、领域或应用场景的局限性。
原文摘要 · Abstract (English)
Drawing inspiration from recent findings including surprisingly decent performance of transformers without positional encoding (NoPE) in the domain of language models and how registers (additional throwaway tokens not tied to input) may improve the performance of large vision transformers (ViTs), we invent and test a variant of ViT called fractal ViT that breaks permutation invariance among the tokens by applying an attention mask between the regular tokens and ``summary tokens'' similar to registers, in isolation or in combination with various positional encodings. These models do not improve upon ViT with registers, highlighting the fact that these findings may be scale, domain, or application-specific.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。