让视觉预测按重要性顺序进行,提升模型泛化能力。
DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecture
- 按视觉重要性逐步预测图像区域,模仿人类感知顺序。
- 在多个任务上优于传统并行预测方法,提升分类与识别性能。
- 适合需要强泛化能力的自监督学习场景,如小样本或跨域任务。
自监督视觉表征学习近年进展表明,基于潜在空间预测的目标对学习可迁移特征有效。例如,基于图像的联合嵌入预测架构(I-JEPA)通过可见上下文预测被遮挡目标区域的潜在嵌入来学习表征。然而,其一次性并行预测所有目标区域,缺乏有意义的预测顺序。受人类视觉感知中选择性、渐进式关注主次线索的启发,我们提出Dseq-JEPA:一种判别性序列联合嵌入预测架构,融合判别性有序序列过程与JEPA式学习目标。具体通过(i)利用注意力导出显著图识别主判别区域作为视觉重要性代理;(ii)按判别性顺序预测后续区域,从而在预训练中诱导从主到次线索的类课程语义演进。在多项任务上的广泛实验——包括图像分类(ImageNet)、细粒度视觉分类(iNaturalist21、CUB、Stanford Cars)、检测/分割(MS-COCO、ADE20K)以及低级推理(CLEVR)——显示,相较于I-JEPA变体,Dseq-JEPA始终能学习更具判别性和泛化性的表征。
原文摘要 · Abstract (English)
Recent advances in self-supervised visual representation learning have demonstrated the effectiveness of predictive latent-space objectives for learning transferable features. In particular, Image-based Joint-Embedding Predictive Architecture (I-JEPA) learns representations by predicting latent embeddings of masked target regions from visible context. However, it predicts target regions in parallel and all at once, lacking ability to order predictions meaningfully. Inspired by human visual perception, which attends selectively and progressively from primary to secondary cues, we propose Dseq-JEPA, a Discriminative Sequential Joint-Embedding Predictive Architecture that bridges latent predictive and autoregressive self-supervised learning. Specifically, Dseq-JEPA integrates a discriminatively ordered sequential process with JEPA-style learning objective. This is achieved by (i) identifying primary discriminative regions using an attention-derived saliency map that serves as a proxy for visual importance, and (ii) predicting subsequent regions in discriminative order, inducing a curriculum-like semantic progression from primary to secondary cues in pre-training. Extensive experiments across tasks -- image classification (ImageNet), fine-grained visual categorization (iNaturalist21, CUB, Stanford Cars), detection/segmentation (MS-COCO, ADE20K), and low-level reasoning (CLEVR) -- show that Dseq-JEPA consistently learns more discriminative and generalizable representations compared to I-JEPA variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。