提出轻量级方法，在部分重建中恢复被丢弃的图像块，提升模型性能且不增加计算负担。
PR-MIM: Delving Deeper into Partial Reconstruction in Masked Image Modeling
- 采用渐进式重建与最远采样策略，高效恢复被丢弃的图像块
- 丢弃50%图像块仍保持ViT-B/16原性能，节省28%计算量和36%内存
- 适用于追求高效预训练的视觉模型开发者
掩码图像建模在表示学习方面取得显著进展，但面临高昂的计算成本。一种省算力的方法是仅重建部分掩码块并丢弃其余，称为部分重建。然而这会降低表示质量。现有方法通过时间冗余（静态图像不可用）或注意力图来判断哪些块可丢弃，但引入额外开销。为此，本文提出渐进式重建策略与最远采样策略，以极轻量方式重建被丢弃的块，使所有掩码块均参与监督，保证充分预训练的同时维持部分重建的降耗优势。我们在多种现有框架上验证了该方法的有效性：当丢弃50%图像块时，ViT-B/16实现无损性能，相比标准MAE减少28% FLOPs和36%内存使用。代码将公开。
原文摘要 · Abstract (English)
Masked image modeling has achieved great success in learning representations but is limited by the huge computational costs. One cost-saving strategy makes the decoder reconstruct only a subset of masked tokens and throw the others, and we refer to this method as partial reconstruction. However, it also degrades the representation quality. Previous methods mitigate this issue by throwing tokens with minimal information using temporal redundancy inaccessible for static images or attention maps that incur extra costs and complexity. To address these limitations, we propose a progressive reconstruction strategy and a furthest sampling strategy to reconstruct those thrown tokens in an extremely lightweight way instead of completely abandoning them. This approach involves all masked tokens in supervision to ensure adequate pre-training, while maintaining the cost-reduction benefits of partial reconstruction. We validate the effectiveness of the proposed method across various existing frameworks. For example, when throwing 50% patches, we can achieve lossless performance of the ViT-B/16 while saving 28% FLOPs and 36% memory usage compared to standard MAE. Our source code will be made publicly available
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。