用CLIP嵌入生成图像，提升多样性且保持真实感。
Hierarchical Text-Conditional Image Generation with CLIP Latents
- 先生成文本对应的CLIP图像嵌入，再解码生成图像。
- 图像多样性显著提升，真实感和描述匹配度损失小。
- 支持零样本语言操控，适合需要风格保持的生成任务。
对比模型如CLIP能够学习到捕捉语义与风格的图像鲁棒表示。为利用这些表示进行图像生成，我们提出两阶段模型：一个先验模型根据文本描述生成CLIP图像嵌入，一个解码器基于该嵌入生成图像。我们证明，显式生成图像表示可在几乎不损失逼真度和文本相似性的前提下提升图像多样性。基于图像表示的解码器还能生成保留语义与风格但细节不同的图像变体。此外，CLIP的联合嵌入空间支持零样本语言引导的图像操作。我们使用扩散模型作为解码器，并对比了自回归与扩散模型作为先验，发现后者计算效率更高且生成质量更优。
原文摘要 · Abstract (English)
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。