改进掩码图像建模的初始性能，无需微调即可用
Beyond [cls]: Exploring the true potential of Masked Image Modeling representations
- 发现[MASK]令牌聚合能力弱，因注意力分布过散
- 提出选择性聚合机制，显著提升零样本表现
- 适合资源有限但想直接使用预训练模型的用户
掩码图像建模（MIM）是视觉自监督学习中一种有前景的方法，但其开箱即用的表现通常低于其他方法。由于需要大量数据、高显存消耗和专业知识，多数用户无法负担微调成本，导致实际应用受限。本文探究了MIM表现不佳的原因：是特征本身较弱，还是使用方式不当？通过详细分析，我们发现MIM中的注意力在众多图像块上几乎均匀分布，导致[cls]令牌无法有效聚合信息。基于此洞察，我们提出选择性聚合机制，更充分地利用块级表示中的语义信息，显著提升了MIM的开箱即用性能。
原文摘要 · Abstract (English)
Masked Image Modeling (MIM) has emerged as a promising approach for Self-Supervised Learning (SSL) of visual representations. However, the out-of-the-box performance of MIMs is typically inferior to competing approaches. Most users cannot afford fine-tuning due to the need for large amounts of data, high GPU consumption, and specialized user knowledge. Therefore, the practical use of MIM representations is limited. In this paper we ask what is the reason for the poor out-of-the-box performance of MIMs. Is it due to weaker features produced by MIM models, or is it due to suboptimal usage? Through detailed analysis, we show that attention in MIMs is spread almost uniformly over many patches, leading to ineffective aggregation by the [cls] token. Based on this insight, we propose Selective Aggregation to better capture the rich semantic information retained in patch tokens, which significantly improves the out-of-the-box performance of MIM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。