arXiv:2205.11487cs.CVcs.LG2022-05NeurIPS被引 8.8k

Imagen用大语言模型实现超逼真文生图,文本理解力更强。

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

论文配图:Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
图 1 · 摘自论文原文
  • 用T5等大语言模型编码文本,提升图像生成质量
  • 在COCO上达7.27的FID新纪录,无需训练于COCO数据
  • 引入DrawBench评测,人类更偏好其图文对齐与画质

我们提出Imagen,一种具有前所未有的逼真度和深层语言理解能力的文生图扩散模型。Imagen利用大型Transformer语言模型在理解文本方面的优势,并结合扩散模型在高保真图像生成上的能力。关键发现是:仅在纯文本语料上预训练的通用大语言模型(如T5)在图像合成中表现惊人有效;增加语言模型规模对样本保真度和图文对齐的提升,远超过增大图像扩散模型的效果。Imagen在未使用COCO数据训练的情况下,达到了7.27的FID新纪录,且人类评估认为其生成样本在图文一致性上可媲美COCO真实数据。为更深入评估文生图模型,我们引入DrawBench——一个全面且具有挑战性的基准测试。通过DrawBench,我们对比了Imagen与VQ-GAN+CLIP、Latent Diffusion Models和DALL-E 2,结果显示人类在并列评估中更偏爱Imagen,在图像质量和图文对齐方面均胜出。

原文摘要 · Abstract (English)

We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.

文生图扩散模型语言理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。