优化提示词，让图像生成模型更擅长创造罕见样本。
Minority-Focused Text-to-Image Generation via Prompt Optimization
- 在线优化提示词，在保持语义的前提下激发目标特征。
- 新方法使罕见图像生成质量显著提升，优于现有采样器。
- 适合需要小众数据增强或创意生成的AI研究者使用。
我们研究了基于预训练文本到图像（T2I）潜空间扩散模型生成少数类样本的问题。在T2I生成中，少数类实例可定义为处于文本条件数据分布低密度区域的样本，对数据增强和创意AI等应用具有重要价值。然而，现有预训练T2I扩散模型主要聚焦高密度区域，这主要是由于引导采样器（如CFG）的影响，而这些采样器对高质量生成至关重要。为此，我们提出一种新框架，以缓解T2I扩散模型对高密度区域的偏倚。首先，我们构建一个在线提示词优化框架，可在推理过程中激发期望属性，同时保持用户提示的语义内容。随后，我们将该通用提示优化器定制为专用求解器，通过引入精心设计的似然目标，促进少数类特征的生成。在多种T2I模型上的大量实验表明，相比现有采样器，该方法显著提升了生成高质量少数类实例的能力。代码已公开于https://github.com/soobin-um/MinorityPrompt。
原文摘要 · Abstract (English)
We investigate the generation of minority samples using pretrained text-to-image (T2I) latent diffusion models. Minority instances, in the context of T2I generation, can be defined as ones living on low-density regions of text-conditional data distributions. They are valuable for various applications of modern T2I generators, such as data augmentation and creative AI. Unfortunately, existing pretrained T2I diffusion models primarily focus on high-density regions, largely due to the influence of guided samplers (like CFG) that are essential for high-quality generation. To address this, we present a novel framework to counter the high-density-focus of T2I diffusion models. Specifically, we first develop an online prompt optimization framework that encourages emergence of desired properties during inference while preserving semantic contents of user-provided prompts. We subsequently tailor this generic prompt optimizer into a specialized solver that promotes generation of minority features by incorporating a carefully-crafted likelihood objective. Extensive experiments conducted across various types of T2I models demonstrate that our approach significantly enhances the capability to produce high-quality minority instances compared to existing samplers. Code is available at https://github.com/soobin-um/MinorityPrompt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。