重新评估视觉Transformer中寄存器的必要性，发现并非所有模型都需添加额外令牌。
Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment
- 通过复现与跨架构测试，检验寄存器对注意力图清晰度的影响
- 在DINO、DeiT3等模型上验证寄存器有效性，但非所有模型均适用
- 揭示原论文术语不一致问题，助力后续研究通用化
训练视觉Transformer（ViT）面临显著挑战，其中之一是注意力图中出现伪影，影响可解释性。Darcet等人（2024）指出，这源于ViT需要存储[CLS] token之外的全局信息，并提出添加空输入令牌（称为寄存器）的新方案，成功消除伪影并提升注意力图清晰度。本文复现了该工作，并在DINO、DINOv2、OpenCLIP和DeiT3等多个模型上评估其结论的普适性。虽然确认了部分关键发现，但结果表明某些主张无法普遍适用于其他模型。此外，我们扩展研究至更小规模模型，探讨模型尺寸的影响。最后，我们厘清原论文中存在的术语不一致问题，并分析其对跨模型推广带来的影响。
原文摘要 · Abstract (English)
Training Vision Transformers (ViTs) presents significant challenges, one of which is the emergence of artifacts in attention maps, hindering their interpretability. Darcet et al. (2024) investigated this phenomenon and attributed it to the need of ViTs to store global information beyond the [CLS] token. They proposed a novel solution involving the addition of empty input tokens, named registers, which successfully eliminate artifacts and improve the clarity of attention maps. In this work, we reproduce the findings of Darcet et al. (2024) and evaluate the generalizability of their claims across multiple models, including DINO, DINOv2, OpenCLIP, and DeiT3. While we confirm the validity of several of their key claims, our results reveal that some claims do not extend universally to other models. Additionally, we explore the impact of model size, extending their findings to smaller models. Finally, we untie terminology inconsistencies found in the original paper and explain their impact when generalizing to a wider range of models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。