用白化CLIP嵌入空间快速估算图像和文本的似然值
Whitened CLIP as a Likelihood Surrogate of Images and Captions
- 通过可逆线性变换对CLIP特征做白化处理，使特征服从标准正态分布
- 白化后特征的对数似然仅需计算其欧氏距离平方，速度极快
- 无需训练、实时可用，适合需要高效似然评估的生成任务
图像的似然估计难以计算，但在诸多应用中极具价值。本文研究利用对比语言-图像预训练模型（CLIP）评估图像与文本的似然。提出一种新方法——白化CLIP（Whitened CLIP），通过对CLIP隐空间进行可逆线性变换，使每个特征均值为0、方差为1，且与其他特征无相关性，实现单位协方差矩阵。我们证明白化后的嵌入统计特性可近似为标准正态分布，因此对数似然可简单通过白化空间中的平方欧氏距离估算。该白化过程完全无需训练，仅需预先计算的白化矩阵，计算效率极高。我们通过若干初步实验验证了该似然评分在图像与文本上的有效性与适用性。
原文摘要 · Abstract (English)
Likelihood approximations for images are not trivial to compute and can be useful in many applications. We examine the use of Contrastive Language-Image Pre-training (CLIP) to assess the likelihood of images and captions. We introduce \textit{Whitened CLIP}, a novel transformation of the CLIP latent space via an invertible linear operation. This transformation ensures that each feature in the embedding space has zero mean, unit standard deviation, and no correlation with all other features, resulting in an identity covariance matrix. We show that the whitened embeddings statistics can be well approximated as a standard normal distribution, thus, the log-likelihood is estimated simply by the square Euclidean norm in the whitened embedding space. The whitening procedure is completely training-free and performed using a pre-computed whitening matrix, hence, is very fast. We present several preliminary experiments demonstrating the properties and applicability of these likelihood scores to images and captions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。