用预训练视觉模型提升GAN的图文生成能力,速度快效果好。
Text-to-Image GAN with Pretrained Representations
- 用多个预训练视觉模型增强判别器,提升场景理解力
- 新设计高容量生成器,FID达5.48(COCO)和9.38(CUB)
- 参数少、训练数据少、推理快,适合实际部署
基于文本描述生成图像受到广泛关注。近期扩散模型和自回归模型虽表现出强大表达能力,但存在推理慢、训练成本高等问题。为在复杂场景下实现更强更快的图文生成,我们提出TIGER,一种融合预训练表示的文本到图像GAN。具体地,提出视觉增强判别器,通过堆叠多个预训练视觉模型获取多样表征,增强场景理解与领域泛化能力;同时设计高容量生成器,包含多个新型高容量融合块(HFBlock),由深融合模块与全局融合模块组成,实现有效图文融合。大量实验表明,TIGER在标准和零样本图文生成任务上均表现优异:在标准任务中,于两个挑战性数据集上分别取得新纪录的FID 5.48(COCO)和9.38(CUB);在零样本任务中,以更少参数、更小训练数据量和更快推理速度达到可比性能。更多实验与分析详见补充材料。
原文摘要 · Abstract (English)
Generating desired images conditioned on given text descriptions has received lots of attention. Recently, diffusion models and autoregressive models have demonstrated their outstanding expressivity and gradually replaced GAN as the favored architectures for text-to-image synthesis. However, they still face some obstacles: slow inference speed and expensive training costs. To achieve more powerful and faster text-to-image synthesis under complex scenes, we propose TIGER, a text-to-image GAN with pretrained representations. To be specific, we propose a vision-empowered discriminator and a high-capacity generator. (i) The vision-empowered discriminator absorbs the complex scene understanding ability and the domain generalization ability from pretrained vision models to enhance model performance. Unlike previous works, we explore stacking multiple pretrained models in our discriminator to collect multiple different representations. (ii) The high-capacity generator aims to achieve effective text-image fusion while increasing the model capacity. The high-capacity generator consists of multiple novel high-capacity fusion blocks (HFBlock). And the HFBlock contains several deep fusion modules and a global fusion module, which play different roles to benefit our model. Extensive experiments demonstrate the outstanding performance of our proposed TIGER both on standard and zero-shot text-to-image synthesis tasks. On the standard text-to-image synthesis task, TIGER achieves state-of-the-art performance on two challenging datasets, which obtain a new FID 5.48 (COCO) and 9.38 (CUB). On the zero-shot text-to-image synthesis task, we achieve comparable performance with fewer model parameters, smaller training data size and faster inference speed. Additionally, more experiments and analyses are conducted in the Supplementary Material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。