通过师生模型协作优化掩码与目标，提升视觉自监督学习效果
The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning
- 用师生模型注意力加权生成协同掩码
- 以双模型输出特征作为解码器目标，提升重建质量
- ViT-base在ImageNet上微调准确率提升至85.7%
掩码自编码器（MAE）在自监督视觉表征学习中表现优异。以往方法多采用自定义掩码（如随机、块状）或基于教师模型（如CLIP）的掩码与目标，但忽略了学生模型对教师的反馈作用。本文提出协同掩码与目标框架（CMT-MAE），通过师生模型注意力的线性融合生成协同掩码，并将两模型输出特征联合作为解码器的目标。该简单有效的框架在ImageNet-1K上预训练，实现最优线性探测与微调性能：使用ViT-base时，微调准确率从原始MAE的83.6%提升至85.7%。
原文摘要 · Abstract (English)
Masked autoencoders (MAE) have recently succeeded in self-supervised vision representation learning. Previous work mainly applied custom-designed (e.g., random, block-wise) masking or teacher (e.g., CLIP)-guided masking and targets. However, they ignore the potential role of the self-training (student) model in giving feedback to the teacher for masking and targets. In this work, we present to integrate Collaborative Masking and Targets for boosting Masked AutoEncoders, namely CMT-MAE. Specifically, CMT-MAE leverages a simple collaborative masking mechanism through linear aggregation across attentions from both teacher and student models. We further propose using the output features from those two models as the collaborative target of the decoder. Our simple and effective framework pre-trained on ImageNet-1K achieves state-of-the-art linear probing and fine-tuning performance. In particular, using ViT-base, we improve the fine-tuning results of the vanilla MAE from 83.6% to 85.7%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。