改进ViT自蒸馏中的随机掩码，提升注意力图质量与下游性能。
Rethinking Random Masking in Self-Distillation on ViT
- 仅对学生模型的全局视图进行随机掩码，保留局部视图和教师视图完整
- 在mini-ImageNet上使用DINO-Tiny，显著提升注意力图的细粒度与鲁棒性
- 适合关注自蒸馏训练效率与特征表示质量的研究者
视觉变压器（ViT）在众多视觉任务中表现出色，其中自蒸馏框架如DINO对此贡献显著。现有方法常采用随机掩码以提高训练效率并引入正则化，但近期研究指出，无差别掩码可能无意中破坏关键语义信息，促使更智能的掩码策略发展。本文探讨了在自蒸馏设置下随机掩码的作用，聚焦于DINO框架。具体而言，我们仅对学生的全局视图施加随机掩码，同时保持学生局部视图及教师全局视图的原始未掩码状态。该设计利用DINO的多视图增强机制，在保留清晰监督信号的同时，通过掩码输入增强模型鲁棒性。我们在mini-ImageNet数据集上使用DINO-Tiny进行评估，结果表明，在这种非对称掩码设置下，注意力图更具鲁棒性和细粒度，最终提升下游任务性能。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have demonstrated remarkable performance across a wide range of vision tasks. In particular, self-distillation frameworks such as DINO have contributed significantly to these advances. Within such frameworks, random masking is often utilized to improve training efficiency and introduce regularization. However, recent studies have raised concerns that indiscriminate random masking may inadvertently eliminate critical semantic information, motivating the development of more informed masking strategies. In this study, we explore the role of random masking in the self-distillation setting, focusing on the DINO framework. Specifically, we apply random masking exclusively to the student's global view, while preserving the student's local views and the teacher's global view in their original, unmasked forms. This design leverages DINO's multi-view augmentation scheme to retain clean supervision while inducing robustness through masked inputs. We evaluate our approach using DINO-Tiny on the mini-ImageNet dataset and show that random masking under this asymmetric setup yields more robust and fine-grained attention maps, ultimately enhancing downstream performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。