提出首个基于概率的图文预训练模型，能捕捉图文间多重对应关系。
Probabilistic Language-Image Pre-Training
- 用概率分布替代确定性嵌入，建模图文间的多对多关系。
- 在百亿级数据上训练，零样本图像分类准确率达74.6%。
- 引入不确定令牌和包含损失，提升模型对模糊性的表达能力。
视觉语言模型（VLMs）将对齐的图文对嵌入到联合空间中，但通常依赖确定性嵌入，假设图像与文本之间存在一对一对应关系。这忽略了现实世界中固有的多对多关系——多个标题可描述同一图像，反之亦然。我们提出概率语言-图像预训练（ProLIP），首个在百亿规模图像-文本数据集上仅使用概率目标进行预训练的模型，实现了强大的零样本性能（例如，ViT-B/16在ImageNet上达到74.6%零样本准确率）。ProLIP通过“不确定令牌”高效估计不确定性，无需额外参数。我们还提出一种新颖的包含损失，强制图像-文本对之间以及原始输入与掩码输入之间的分布包含关系。实验表明，利用不确定性估计，ProLIP在下游任务中表现更优，并符合直观的不确定性认知，例如较短文本更不确定，更通用输入包含具体输入。利用文本不确定性，我们在少样本设置下将ImageNet准确率从74.6%提升至75.8%，验证了概率方法的实际优势。代码已开源。
原文摘要 · Abstract (English)
Vision-language models (VLMs) embed aligned image-text pairs into a joint space but often rely on deterministic embeddings, assuming a one-to-one correspondence between images and texts. This oversimplifies real-world relationships, which are inherently many-to-many, with multiple captions describing a single image and vice versa. We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability (e.g., 74.6% ImageNet zero-shot accuracy with ViT-B/16). ProLIP efficiently estimates uncertainty by an "uncertainty token" without extra parameters. We also introduce a novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs. Experiments demonstrate that, by leveraging uncertainty estimates, ProLIP benefits downstream tasks and aligns with intuitive notions of uncertainty, e.g., shorter texts being more uncertain and more general inputs including specific ones. Utilizing text uncertainties, we further improve ImageNet accuracy from 74.6% to 75.8% (under a few-shot setting), supporting the practical advantages of our probabilistic approach. The code is available at https://github.com/naver-ai/prolip
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。