通过遮挡生成语义差异视图，实现高效视觉表征预训练
One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation
- 随机遮挡图像块生成语义级差异视图，避免实例级重复
- 在4张A100上133小时完成ViT-L/16预训练，下游准确率达85.8%
- 无需人工数据增强或额外模块，适合高效视觉建模研究
本文提出一种可扩展且简洁的视觉概念表征预训练范式——遮挡图像对比学习（OCL）。其核心思想是：在单张图像内随机遮挡补丁以生成不同视图，并在小批量图像间进行对比学习。两个关键设计为：第一，遮挡能有效减少图像固有的概念冗余，生成细粒度语义差异；第二，对比学习擅长提取高层语义特征，避免图像重建带来的高频干扰和额外开销。重要的是，OCL无需依赖人工数据增强或额外辅助模块，即可高效学习高语义概念表征。实验表明，该方法对视觉变换器具有强可扩展性：使用4张A100 GPU，ViT-L/16可在133小时内完成预训练，在下游微调任务中达到85.8%准确率。代码已公开于https://anonymous.4open.science/r/OLRS/。
原文摘要 · Abstract (English)
This paper proposes a scalable and straightforward pre-training paradigm for efficient visual conceptual representation called occluded image contrastive learning (OCL). Our OCL approach is simple: we randomly mask patches to generate different views within an image and contrast them among a mini-batch of images. The core idea behind OCL consists of two designs. First, masked tokens have the potential to significantly diminish the conceptual redundancy inherent in images, and create distinct views with substantial fine-grained differences on the semantic concept level instead of the instance level. Second, contrastive learning is adept at extracting high-level semantic conceptual features during the pre-training, circumventing the high-frequency interference and additional costs associated with image reconstruction. Importantly, OCL learns highly semantic conceptual representations efficiently without relying on hand-crafted data augmentations or additional auxiliary modules. Empirically, OCL demonstrates high scalability with Vision Transformers, as the ViT-L/16 can complete pre-training in 133 hours using only 4 A100 GPUs, achieving 85.8\% accuracy in downstream fine-tuning tasks. Code is available at https://anonymous.4open.science/r/OLRS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。