用标签引导对比学习提升文本图像生成质量
A Framework For Image Synthesis Using Supervised Contrastive Learning
- 引入标签引导的对比学习，同时利用跨模态和同模态语义关系
- 在COCO数据集上使FID降低最高达30.1%（AttnGAN）
- 适合关注生成质量提升的图像生成研究者
文本到图像（T2I）生成旨在根据文本描述生成逼真图像。生成对抗网络（GAN）在此任务中表现优异。典型的T2I GAN采用两阶段方法：先基于对齐的图文配对预训练跨模态表示，再基于此训练图像生成器。然而，这类表示忽略了同模态内部的语义对应关系（如相同标签的图像）。语义标签事先描述了图像内在分布模式及图像间的潜在关联，是对文本描述的补充，有助于全面理解图像特征。本文提出一种框架，通过标签引导的监督对比学习，同时利用跨模态与同模态对应关系。将T2I GAN扩展为预训练和生成阶段共享参数的双对比分支，有效聚类语义相似的图文对表示，从而提升生成图像质量。我们在四个新型T2I GAN上进行验证，使用单对象数据集CUB和多对象数据集COCO，显著提升了图像生成评估指标的Inception Score（IS）和Frechet Inception Distance（FID）。尤其在复杂多对象的COCO数据集上，我们的框架使AttnGAN、DM-GAN、SSA-GAN和GALIP的FID分别降低30.1%、27.3%、16.2%和17.1%。与其它标签引导的T2I GAN相比，结果也证明了本方法的有效性与竞争力。
原文摘要 · Abstract (English)
Text-to-image (T2I) generation aims at producing realistic images corresponding to text descriptions. Generative Adversarial Network (GAN) has proven to be successful in this task. Typical T2I GANs are 2 phase methods that first pretrain an inter-modal representation from aligned image-text pairs and then use GAN to train image generator on that basis. However, such representation ignores the inner-modal semantic correspondence, e.g. the images with same label. The semantic label in priory describes the inherent distribution pattern with underlying cross-image relationships, which is supplement to the text description for understanding the full characteristics of image. In this paper, we propose a framework leveraging both inter- and inner-modal correspondence by label guided supervised contrastive learning. We extend the T2I GANs to two parameter-sharing contrast branches in both pretraining and generation phases. This integration effectively clusters the semantically similar image-text pair representations, thereby fostering the generation of higher-quality images. We demonstrate our framework on four novel T2I GANs by both single-object dataset CUB and multi-object dataset COCO, achieving significant improvements in the Inception Score (IS) and Frechet Inception Distance (FID) metrics of imagegeneration evaluation. Notably, on more complex multi-object COCO, our framework improves FID by 30.1%, 27.3%, 16.2% and 17.1% for AttnGAN, DM-GAN, SSA-GAN and GALIP, respectively. We also validate our superiority by comparing with other label guided T2I GANs. The results affirm the effectiveness and competitiveness of our approach in advancing the state-of-the-art GAN for T2I generation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。