arXiv:2501.01368cs.CV2025-01

无需微调,即可在测试时精准控制图像生成的复杂空间条件。

Test-time Controllable Image Generation by Explicit Spatial Constraint Enforcement

  • 分离语义与几何条件,分别在生成中强制一致性。
  • 在Coco-stuff上相比顶尖无训练方法布局一致率提升30%。
  • 适合需要灵活控制生成布局的视觉应用开发者。

近期文本到图像生成广泛采用各类空间条件,如掩码、边界框和关键点。然而,多数现有方法需针对特定条件微调模型,导致测试时泛化能力差。现有无训练方法仅对简化提示和条件有效。本文提出一种新颖且通用的测试时可控生成方法,支持自然文本提示与复杂条件。我们首先将空间条件解耦为语义与几何两类,并在生成过程中分别强制其一致性。针对语义条件,通过补全提示并量化注意力图中分散词的统计特征与词空间距离,消除干扰。针对复杂几何条件,设计几何变换模块:在注意力图中识别感兴趣区域(RoI),并基于该区域调整类别相关潜在表示。更重要的是,提出基于扩散的潜变量重填方法,显式移除RoI区域的潜在影响,减少生成图像中的伪影。在Coco-stuff数据集上的实验表明,相较最先进无训练方法,布局一致性评估指标相对提升30%。

原文摘要 · Abstract (English)

Recent text-to-image generation favors various forms of spatial conditions, e.g., masks, bounding boxes, and key points. However, the majority of the prior art requires form-specific annotations to fine-tune the original model, leading to poor test-time generalizability. Meanwhile, existing training-free methods work well only with simplified prompts and spatial conditions. In this work, we propose a novel yet generic test-time controllable generation method that aims at natural text prompts and complex conditions. Specifically, we decouple spatial conditions into semantic and geometric conditions and then enforce their consistency during the image-generation process individually. As for the former, we target bridging the gap between the semantic condition and text prompts, as well as the gap between such condition and the attention map from diffusion models. To achieve this, we propose to first complete the prompt w.r.t. semantic condition, and then remove the negative impact of distracting prompt words by measuring their statistics in attention maps as well as distances in word space w.r.t. this condition. To further cope with the complex geometric conditions, we introduce a geometric transform module, in which Region-of-Interests will be identified in attention maps and further used to translate category-wise latents w.r.t. geometric condition. More importantly, we propose a diffusion-based latents-refill method to explicitly remove the impact of latents at the RoI, reducing the artifacts on generated images. Experiments on Coco-stuff dataset showcase 30$\%$ relative boost compared to SOTA training-free methods on layout consistency evaluation metrics.

图像生成扩散模型空间控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。