让AI画画更听懂复杂指令,推理时自动修正偏差。
Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time

- 用文本对齐的初始噪声替代随机噪声,提升生成起点准确性。
- 引入自检自纠机制,在生成过程中动态修正语义错误。
- 无需重新训练,适合希望提高绘图精准度的用户使用。
尽管文生图扩散模型能生成高质量图像,但在处理复杂组合提示时常出现与文本不一致的问题。本文提出一种无需训练的框架 AnchorSteer,同时优化生成初始状态和去噪过程。首先通过 CLIP 提取先验信息,结合新型潜空间先验蒸馏采样(LP-SDS)目标,生成语义对齐的初始噪声;其次设计“思考-擦除-重绘”循环,利用视觉语言模型诊断语义偏差,并针对性地修正潜在错误内容或恢复缺失属性。在 GenEval 与 T2I-CompBench++ 数据集上的实验表明,AnchorSteer 在保持高视觉质量的同时,显著提升了图文一致性,优于现有各类无训练方法。
原文摘要 · Abstract (English)
While text-to-image diffusion models achieve impressive visual quality, they frequently struggle to maintain precise alignment with complex compositional prompts. An effective strategy is to improve the inference process of diffusion models, thereby better leveraging their pretrained priors to address misalignment. Existing training-free methods can be divided into two categories. The first category focuses on improving the randomly sampled initial noise, either performing costly search over noise pools or manipulating sampled noise without ensuring reliable semantic injection. The second category focuses on improving the denoising trajectory, lacking explicit mechanisms to timely diagnose and correct semantic errors. we propose \textbf{AnchorSteer}, a training-free framework that exerts fine-grained control over \textbf{both initialization} and \textbf{the denoising trajectory}. AnchorSteer consists of two synergistic components: \textbf{Semantic Anchoring} replaces uninformative Gaussian noise with text-aligned initializations via CLIP-based prior extraction and a novel Latent-Prior Score Distillation Sampling (LP-SDS) objective. Specifically, LP-SDS distills CLIP visual priors into the knowledge distribution of diffusion models, mitigating the domain gap between CLIP-based priors and diffusion-based priors. \textbf{Reflective Steering} transforms passive denoising with an active Think--Erase--Retouch loop that enables mid-generation self-correction. It leverages VLM-based diagnosis to detect semantic deviations and performs targeted latent refinement to suppress erroneous content and recover missing attributes. Extensive experiments on GenEval and T2I-CompBench++ demonstrate that AnchorSteer consistently outperforms existing baselines in text--image alignment while preserving high visual quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。