通过可微嵌入优化,让文本生成图像更贴合语义与审美。
IPGO: Indirect Prompt Gradient Optimization for Parameter-Efficient Prompt-level Fine-Tuning on Text-to-Image Models
- 在提示词首尾注入可微嵌入,用低秩旋转结构实现梯度优化。
- 提示词级优化胜率超99%,批量优化达75%胜率,超越当前最佳方法。
- 仅需少量数据和资源,适合快速提升生成质量的开发者使用。
文生图扩散模型虽能根据文本生成图像,但常存在语义、美学及人类偏好对齐不佳的问题。本文提出一种参数高效的新框架IPGO,通过在提示词嵌入前后注入连续可微嵌入,并利用低秩结构结合旋转带来的灵活性与非线性,实现对注入嵌入的梯度优化。该方法在范围、正交性和一致性约束下缩小搜索空间,促进稳定解并确保注入嵌入与原始提示的一致性。其扩展IPGO+在提示嵌入上加入无参数交叉注意力机制,强化原始提示与插入嵌入间的依赖关系。我们在三个不同复杂度的数据集上,基于图像美学、图文对齐与人类偏好三种奖励模型,进行了提示词级(IPGO)与提示批处理级(IPGO+)的广泛评估。结果表明,IPGO在提示词级学习中胜率超过99%,显著优于SOTA基准(包括Stable Diffusion v1.5原提示、TextCraftor等文本嵌入方法,DRaFT、DDPO等训练型方法,以及DPO-Diffusion、Promptist、ChatGPT-4o等免训练方法)。IPGO+在提示批处理学习中表现相当甚至更优,胜率达75%。此外,实验验证了IPGO的泛化能力,能在极低数据与资源消耗下显著提升图像质量。
原文摘要 · Abstract (English)
Text-to-Image Diffusion models excel at generating images from text prompts but often exhibit suboptimal alignment with content semantics, aesthetics, and human preferences. To address these limitations, this study proposes a novel parameter-efficient framework, Indirect Prompt Gradient Optimization (IPGO), for prompt-level diffusion model fine-tuning. IPGO enhances prompt embeddings by injecting continuously differentiable embeddings at the beginning and end of the prompt embeddings, leveraging low-rank structures with the flexibility and nonlinearity from rotations. This approach enables gradient-based optimization of injected embeddings under range, orthonormality, and conformity constraints, effectively narrowing the search space, promoting a stable solution, and ensuring alignment between the embeddings of the injected embeddings and the original prompt. Its extension IPGO+ adds a parameter-free cross-attention mechanism on the prompt embedding to enforce dependencies between the original prompt and the inserted embeddings. We conduct extensive evaluations through prompt-wise (IPGO) and prompt-batch (IPGO+) training using three reward models of image aesthetics, image-text alignment, and human preferences across three datasets of varying complexity. The results show that IPGO consistently outperforms SOTA benchmarks, including stable diffusion v1.5 with raw prompts, text-embedding-based methods (TextCraftor), training-based methods (DRaFT and DDPO), and training-free methods (DPO-Diffusion, Promptist, and ChatGPT-4o). Specifically, IPGO achieves a win-rate exceeding 99% in prompt-wise learning, and IPGO+ achieves a comparable, but often better performance against current SOTAs (a 75% win rate) in prompt-batch learning. Moreover, we illustrate IPGO's generalizability and its capability to significantly enhance image quality while requiring minimal data and resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。