通过提示感知的噪声投影,提升扩散模型生成图像与提示的一致性。
Noise Projection: Closing the Prompt-Agnostic Gap Behind Text-to-Image Misalignment in Diffusion Models
- 用提示条件化噪声投影,使初始噪声更符合训练分布。
- 在多个提示下显著提升图文对齐效果,优于多采样后选择方法。
- 无需参考图或手工先验,单次前向传播即可完成优化。
在文本到图像生成中,不同的初始噪声会引导预训练的Stable Diffusion(SD)模型走不同的去噪路径。尽管这能产生多样化图像,但部分结果可能与提示不一致。现有方法要么改变去噪动态,要么多次采样后进行后选。本文将错位归因于训练-推理不匹配:训练时提示条件噪声位于提示特定的潜空间子集,而推理时噪声来自提示无关的高斯先验。为此,提出一种噪声投影器,在去噪前对初始噪声进行提示条件化修正。基于提示嵌入,该投影器将噪声映射为更符合SD训练期间观察到分布的提示感知版本,且不修改原始SD模型。框架包含三步:首先采样若干噪声并从视觉语言模型(VLM)获取对应图像的词级反馈;然后将这些信号蒸馏为奖励模型;最后通过准直接偏好优化(quasi-direct preference optimization)优化噪声投影器。设计具有两大优势:(i) 无需参考图像或手工先验;(ii) 推理开销小,以单次前向传播替代多样本选择。大量实验表明,该提示感知噪声投影在多种提示下均显著提升图文对齐能力。
原文摘要 · Abstract (English)
In text-to-image generation, different initial noises induce distinct denoising paths with a pretrained Stable Diffusion (SD) model. While this pattern could output diverse images, some of them may fail to align well with the prompt. Existing methods alleviate this issue either by altering the denoising dynamics or by drawing multiple noises and conducting post-selection. In this paper, we attribute the misalignment to a training-inference mismatch: during training, prompt-conditioned noises lie in a prompt-specific subset of the latent space, whereas at inference the noise is drawn from a prompt-agnostic Gaussian prior. To close this gap, we propose a noise projector that applies text-conditioned refinement to the initial noise before denoising. Conditioned on the prompt embedding, it maps the noise to a prompt-aware counterpart that better matches the distribution observed during SD training, without modifying the SD model. Our framework consists of these steps: we first sample some noises and obtain token-level feedback for their corresponding images from a vision-language model (VLM), then distill these signals into a reward model, and finally optimize the noise projector via a quasi-direct preference optimization. Our design has two benefits: (i) it requires no reference images or handcrafted priors, and (ii) it incurs small inference cost, replacing multi-sample selection with a single forward pass. Extensive experiments further show that our prompt-aware noise projection improves text-image alignment across diverse prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。