不用对比学习，靠预测文本嵌入实现视觉语言对齐。
Non-Contrastive Vision-Language Learning with Predictive Embedding Alignment
- 用图像增强视图预测文本嵌入，对齐视觉与文本表示。
- 在三个基准数据集上零样本分类表现超越多个基线模型。
- 训练更稳定，仅需一个超参数，适合快速实验与部署。
视觉语言模型已革新多模态表征学习，但主流对比方法（如CLIP）需大批次、精心设计的负样本及大量超参数调优。本文提出NOVA——一种基于联合嵌入预测与分布正则化的非对比视觉语言对齐框架。NOVA通过从图像增强视图预测文本嵌入，将视觉表征对齐至冻结的领域特定文本编码器（使用ClinicalBERT），同时通过简化的各向同性高斯正则化（SIGReg）强制嵌入呈各向同性高斯分布。该方法无需负样本、动量编码器或梯度截断，训练目标简化为单个超参数。在使用MIMIC-CXR训练的视觉变换器与ClinicalBERT组合下，于零样本胸部X光分类任务中，在三个基准数据集上均优于多个标准基线，并表现出显著更一致的训练过程。结果表明，非对比预训练提供了一种更简单、更稳定、更有效的替代方案。
原文摘要 · Abstract (English)
Vision-language models have transformed multimodal representation learning, yet dominant contrastive approaches like CLIP require large batch sizes, careful negative sampling, and extensive hyperparameter tuning. We introduce NOVA, a NOn-contrastive Vision-language Alignment framework based on joint embedding prediction with distributional regularization. NOVA aligns visual representations to a frozen, domain-specific text encoder by predicting text embeddings from augmented image views, while enforcing an isotropic Gaussian structure via Sketched Isotropic Gaussian Regularization (SIGReg). This eliminates the need for negative sampling, momentum encoders, or stop-gradients, reducing the training objective to a single hyperparameter. We evaluate NOVA on zeroshot chest X-ray classification using ClinicalBERT as the text encoder and Vision Transformers trained from scratch on MIMIC-CXR. On zero-shot classification across three benchmark datasets, NOVA outperforms multiple standard baselines while exhibiting substantially more consistent training runs. Our results demonstrate that non-contrastive vision-language pretraining offers a simpler, more stable, and more effective alternative to contrastive methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。