用CLIP嵌入的舒尔补计算文本引导生成的多样性,更准确评估图像生成模型的创造力。
Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings
- 通过舒尔补分解图像-文本协方差矩阵,分离出与提示相关的多样性成分。
- 提出Scendi分数,量化同一提示下生成图像的内在多样性,数值实证有效。
- 可屏蔽特定提示影响,聚焦或忽略图像中特定物体的嵌入特征,适合模型优化研究者。
利用CLIP嵌入评估文本到图像生成模型输出保真度已广泛研究。尽管广泛采用的CLIPScore基于文本与图像嵌入的余弦相似度,能有效衡量生成图像与提示的对齐程度,但无法量化模型在相同提示下生成图像的多样性。本文将CLIP嵌入拓展用于量化和解释文本到图像模型的内在多样性——即对相似提示生成多样化图像的能力,称为提示感知多样性。为此,我们提出将基于CLIP的图像数据核协方差矩阵分解为基于文本和非文本成分。通过联合图像-文本核协方差矩阵的舒尔补进行分解,并将分解后成分的矩阵熵定义为舒尔补熵多样性(Scendi)分数,作为提示引导生成模型的提示感知多样性度量。此外,我们讨论了基于舒尔补的分解在消除给定提示对图像CLIP嵌入影响中的应用,实现对嵌入向量中特定物体的关注或去关注。我们展示了若干数值结果,将所提出的Scendi分数应用于文本到图像及大语言模型(文本到文本)的评估。数值结果表明,Scendi分数成功捕捉了提示引导生成模型的内在多样性。代码库位于 https://github.com/aziksh-ospanov/scendi-score。
原文摘要 · Abstract (English)
The use of CLIP embeddings to assess the fidelity of samples produced by text-to-image generative models has been extensively explored in the literature. While the widely adopted CLIPScore, derived from the cosine similarity of text and image embeddings, effectively measures the alignment of a generated image, it does not quantify the diversity of images generated by a text-to-image model. In this work, we extend the application of CLIP embeddings to quantify and interpret the intrinsic diversity of text-to-image models, which are responsible for generating diverse images from similar text prompts, which we refer to as prompt-aware diversity. To achieve this, we propose a decomposition of the CLIP-based kernel covariance matrix of image data into text-based and non-text-based components. Using the Schur complement of the joint image-text kernel covariance matrix, we perform this decomposition and define the matrix-based entropy of the decomposed component as the Schur Complement ENtopy DIversity (Scendi) score, as a measure of the prompt-aware diversity for prompt-guided generative models. Additionally, we discuss the application of the Schur complement-based decomposition to nullify the influence of a given prompt on the CLIP embedding of an image, enabling focus or defocus of the embedded vectors on specific objects. We present several numerical results that apply our proposed Scendi score to evaluate text-to-image and LLM (text-to-text) models. Our numerical results indicate the success of the Scendi score in capturing the intrinsic diversity of prompt-guided generative models. The codebase is available at https://github.com/aziksh-ospanov/scendi-score.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。