提出迭代优化方法AIR,提升零样本生成模型的图像质量。
AIR: Zero-shot Generative Model Adaptation with Iterative Refinement
- 基于文本与图像偏移不一致的发现,设计迭代优化策略
- 在26个实验中实现当前最优生成质量
- 适合需要高质量零样本生成的研究者使用
零样本生成模型适应(ZSGM)旨在仅通过文本指导而无需目标域样本,将预训练生成器适配到新领域。现有方法依赖方向性损失,使图像偏移与文本偏移在CLIP嵌入空间对齐,类似自然语言中的类比推理。然而,这类方法假设图像与文本偏移完全对齐,导致生成图像质量下降。本文首次系统分析了多个公开数据集下CLIP中文本偏移与图像偏移的错位现象,发现概念距离越近,偏移错位越小。基于此,提出自适应迭代精炼(AIR),首个聚焦于提升目标域图像质量的ZSGM方法。在26组实验中,定性、定量及用户研究均表明AIR达到当前最优性能,更多实验见补充材料。
原文摘要 · Abstract (English)
Zero-shot generative model adaptation (ZSGM) aims to adapt a pre-trained generator to a target domain using only text guidance and without any samples from the target domain. Central to recent ZSGM approaches are directional loss which use the text guidance in the form of aligning the image offset with text offset in the embedding space of a vision-language model like CLIP. This is similar to the analogical reasoning in NLP where the offset between one pair of words is used to identify a missing element in another pair by aligning the offset between these two pairs. However, a major limitation of existing ZSGM methods is that the learning objective assumes the complete alignment between image offset and text offset in the CLIP embedding space, resulting in quality degrade in generated images. Our work makes two main contributions. Inspired by the offset misalignment studies in NLP, as our first contribution, we perform an empirical study to analyze the misalignment between text offset and image offset in CLIP embedding space for various large publicly available datasets. Our important finding is that offset misalignment in CLIP embedding space is correlated with concept distance, i.e., close concepts have a less offset misalignment. To address the limitations of the current approaches, as our second contribution, we propose Adaptation with Iterative Refinement (AIR) which is the first ZSGM approach to focus on improving target domain image quality based on our new insight on offset misalignment.Qualitative, quantitative, and user study in 26 experiment setups consistently demonstrate the proposed AIR approach achieves SOTA performance. Additional experiments are in Supp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。