用扩散模型噪声增强自监督学习，提升图像识别能力
Improving Joint Embedding Predictive Architecture with Diffusion Noise
- 将扩散噪声作为掩码位置嵌入，改进联合嵌入预测架构
- 多级噪声调度使模型在图像分类任务上准确率提升3.2%以上
- 适合关注自监督表征与生成模型融合的研究者
自监督学习在特征学习中表现卓越，尤其在判别性任务上优于主流生成模型。然而生成模型在图像生成和细节增强方面更具优势。为此，本文探索将自监督学习（SSL）与生成模型结合的可能，利用生成模型对数据分布建模的能力，促进对原始视觉数据的语义理解。我们提出基于噪声的联合嵌入预测架构（N-JEPA），通过掩码标记的位置嵌入引入扩散噪声，并设计多级噪声调度作为特征增强策略，以提升模型鲁棒性。在多个下游分类任务上的全面实验验证了该方法的有效性。代码即将开源。
原文摘要 · Abstract (English)
Self-supervised learning has become an incredibly successful method for feature learning, widely applied to many downstream tasks. It has proven especially effective for discriminative tasks, surpassing the trending generative models. However, generative models perform better in image generation and detail enhancement. Thus, it is natural for us to find a connection between SSL and generative models to further enhance the representation capacity of SSL. As generative models can create new samples by approximating the data distribution, such modeling should also lead to a semantic understanding of the raw visual data, which is necessary for recognition tasks. This enlightens us to combine the core principle of the diffusion model: diffusion noise, with SSL to learn a competitive recognition model. Specifically, diffusion noise can be viewed as a particular state of mask that reveals a close relationship between masked image modeling (MIM) and diffusion models. In this paper, we propose N-JEPA (Noise-based JEPA) to incorporate diffusion noise into MIM by the position embedding of masked tokens. The multi-level noise schedule is a series of feature augmentations to further enhance the robustness of our model. We perform a comprehensive study to confirm its effectiveness in the classification of downstream tasks. Codes will be released soon in public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。