首个无需负样本的端到端视觉语言预训练方法,提升密集语义特征表现。
LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
- 采用跨模态预测与分布正则化,无负样本、无温度参数等复杂设计。
- 在多个下游任务中优于对比基线,尤其在密集预测任务上显著领先。
- 适合构建冻结视觉主干的多模态模型,对语义分割等任务特别有效。
视觉语言预训练仍以对比学习为主,而视觉自监督学习已转向非对比方法。同时,视觉语言编码器的角色转变:不再仅用于零样本分类,而是作为视觉语言模型和密集预测系统的冻结视觉主干,使用完整的补丁网格而非单个池化嵌入。我们提出LeVLJEPA,首个完全非对比的端到端视觉语言预训练方法。该方法通过带停止梯度目标的跨模态预测与每模态分布正则化进行学习,无需负样本、温度参数、动量编码器或师生训练调度,在大规模训练下稳定收敛。结果表明,该编码器生成的密集语义特征显著更强:作为冻结的视觉语言模型主干,在GQA、VQAv2和POPE三个数据集上均超越对比基线,且在语义分割任务上表现更优,而在全局读出任务如线性探查上保持相当水平。这些结果确立了非对比预训练在生成密集语义视觉特征方面的有效性。
原文摘要 · Abstract (English)
Vision-language pretraining remains dominated by contrastive objectives, whereas vision-only self-supervised learning has largely adopted non-contrastive methods. At the same time, the role of vision-language encoders has shifted: they are increasingly deployed not as zero-shot classifiers but as the frozen visual backbone of vision-language models and dense prediction systems, which consume the full grid of patch tokens rather than a single pooled embedding. We introduce LeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. LeVLJEPA learns through cross-modal prediction with stop-gradient targets and per-modality distributional regularization, without negatives, temperature, momentum encoder, or teacher-student schedule, and trains stably at large scale. We find that the resulting encoder provides markedly stronger dense semantic features for downstream use: as a frozen vision-language-model backbone, LeVLJEPA is the strongest of the evaluated encoders across GQA, VQAv2, and POPE under two distinct language models, and outperforms contrastive baselines on semantic segmentation, while remaining on par on global readouts such as linear probing. These results establish non-contrastive pretraining as an effective means of producing dense semantic vision features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。