对比两个文生图模型，找出它们生成图像的差异及触发原因。
Discovering Divergent Representations between Text-to-Image Models
- 用进化搜索算法发现一个模型生成而另一个不生成的视觉属性。
- 在60个输入相关差异上验证，效果优于基于LLM和VLM的基线方法。
- 可揭示模型偏见，适合关注生成模型差异与公平性的研究者。
本文研究两个生成模型在何时以及如何产生不同的视觉表征。给定两个文生图模型，目标是发现一个模型输出中出现但另一个未出现的视觉属性，以及触发这些差异的提示类型。例如，表达强烈情绪的提示可能使一个模型生成火焰，而另一个则不会。为此，我们提出CompCon（比较概念）算法，一种进化搜索方法，能发现某一模型输出中更显著的视觉属性，并关联其对应的提示概念。为评估该方法，我们构建了自动化数据生成流水线，创建了包含60个输入依赖差异的ID2数据集，并与多种基于LLM和VLM的基线进行对比。最后，使用CompCon对比多个主流文生图模型，发现如PixArt在涉及孤独的提示中常生成湿漉漉的街道，而Stable Diffusion 3.5在涉及非裔美国人媒体职业时则表现出不同特征。代码已开源。
原文摘要 · Abstract (English)
In this paper, we investigate when and how visual representations learned by two different generative models diverge. Given two text-to-image models, our goal is to discover visual attributes that appear in images generated by one model but not the other, along with the types of prompts that trigger these attribute differences. For example, "flames" might appear in one model's outputs when given prompts expressing strong emotions, while the other model does not produce this attribute given the same prompts. We introduce CompCon (Comparing Concepts), an evolutionary search algorithm that discovers visual attributes more prevalent in one model's output than the other, and uncovers the prompt concepts linked to these visual differences. To evaluate CompCon's ability to find diverging representations, we create an automated data generation pipeline to produce ID2, a dataset of 60 input-dependent differences, and compare our approach to several LLM- and VLM-powered baselines. Finally, we use CompCon to compare popular text-to-image models, finding divergent representations such as how PixArt depicts prompts mentioning loneliness with wet streets and Stable Diffusion 3.5 depicts African American people in media professions. Code at: https://github.com/adobe-research/CompCon
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。