提出CoMA框架，用互补掩码和动态窗口注意力提升图像预训练效率。
CoMA: Complementary Masking and Hierarchical Dynamic Multi-Window Self-Attention in a Unified Pre-training Framework
- 采用互补掩码策略，均匀覆盖所有像素，提升特征学习效率。
- 仅用12%训练轮次即达MAE性能，每轮训练时间减少10%。
- 适合追求高效预训练的视觉模型研发者使用。
掩码自编码器（MAE）通过随机移除部分视觉标记并重建原图来实现图像表征的自监督学习，显著提升预训练效率并在下游任务中表现出色。然而，采用随机掩码的MAE类方法通常需要更多预训练轮次以维持适应性；同时，传统ViT在多层间固定空间分辨率，导致参数利用效率低。为此，本文提出互补掩码自编码器（CoMA），通过互补掩码策略确保所有像素被均匀采样，从而增强全局特征学习能力与模型适应性。此外，引入层次化动态多窗自注意力（DyViT），大幅降低参数量与浮点运算量，同时提升细粒度特征学习能力。在ImageNet-1K上基于CoMA预训练的DyViT，仅需MAE 12%的预训练轮次即可达到相当的下游性能，并实现每轮训练时间减少10%，充分验证其优越的预训练效率。
原文摘要 · Abstract (English)
Masked Autoencoders (MAE) achieve self-supervised learning of image representations by randomly removing a portion of visual tokens and reconstructing the original image as a pretext task, thereby significantly enhancing pretraining efficiency and yielding excellent adaptability across downstream tasks. However, MAE and other MAE-style paradigms that adopt random masking generally require more pre-training epochs to maintain adaptability. Meanwhile, ViT in MAE suffers from inefficient parameter use due to fixed spatial resolution across layers. To overcome these limitations, we propose the Complementary Masked Autoencoders (CoMA), which employ a complementary masking strategy to ensure uniform sampling across all pixels, thereby improving effective learning of all features and enhancing the model's adaptability. Furthermore, we introduce DyViT, a hierarchical vision transformer that employs a Dynamic Multi-Window Self-Attention (DM-MSA), significantly reducing the parameters and FLOPs while improving fine-grained feature learning. Pre-trained on ImageNet-1K with CoMA, DyViT matches the downstream performance of MAE using only 12% of the pre-training epochs, demonstrating more effective learning. It also attains a 10% reduction in pre-training time per epoch, further underscoring its superior pre-training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。