通过反向生成提示词,从图像还原创作思路并生成新图。
Reverse Prompt: Cracking the Recipe Inside Text-to-Image Generation
- 用迭代梯度优化法自动改进初始提示词,逼近参考图像。
- 在多个数据集上快速收敛,生成的提示词与原图CLIP相似度高。
- 适合想理解艺术创作逻辑或快速生成多样化图像的研究者。
文本到图像生成日益流行,但生成理想图像常需大量提示工程。本文探索从参考图像中解码文本提示,即图像反向提示工程,以洞察参考图像的创作思路,理解优秀艺术家的创作过程,并生成新图像。为此,我们提出自动反向提示优化(ARPO)方法:通过迭代的仿射梯度提示优化流程,将初始提示逐步优化为高质量提示。具体包括:1)从当前提示生成复现图像,体现其引导能力;2)生成文本梯度(候选提示),以缩小复现图像与参考图像的差异;3)使用贪婪搜索更新当前提示,最大化提示与参考图像间的CLIP相似度。实验对比了手写技巧、基于梯度的提示调优、图像描述生成和数据驱动选择等基线方法。定量与定性结果表明,ARPO能快速收敛生成高质量反向提示。更重要的是,直接编辑这些反向提示即可轻松生成风格多样、内容新颖的图像。代码将公开。
原文摘要 · Abstract (English)
Text-to-image generation has become increasingly popular, but achieving the desired images often requires extensive prompt engineering. In this paper, we explore how to decode textual prompts from reference images, a process we refer to as image reverse prompt engineering. This technique enables us to gain insights from reference images, understand the creative processes of great artists, and generate impressive new images. To address this challenge, we propose a method known as automatic reverse prompt optimization (ARPO). Specifically, our method refines an initial prompt into a high-quality prompt through an iteratively imitative gradient prompt optimization process: 1) generating a recreated image from the current prompt to instantiate its guidance capability; 2) producing textual gradients, which are candidate prompts intended to reduce the difference between the recreated image and the reference image; 3) updating the current prompt with textual gradients using a greedy search method to maximize the CLIP similarity between prompt and reference image. We compare ARPO with several baseline methods, including handcrafted techniques, gradient-based prompt tuning methods, image captioning, and data-driven selection method. Both quantitative and qualitative results demonstrate that our ARPO converges quickly to generate high-quality reverse prompts. More importantly, we can easily create novel images with diverse styles and content by directly editing these reverse prompts. Code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。