arXiv:2412.01284cs.CVcs.AI2024-12被引 2

无需掩码和重训练,实现图像中物体位置的精确控制。

MFTF: Mask-free Training-free Object Level Layout Control Diffusion Model

  • 通过动态生成注意力掩码,分离并调整目标物体
  • 支持单/多物体的位置变换,精度高且无需额外输入
  • 适合需要精细布局编辑的视觉内容创作场景

文本到图像生成模型已革新内容创作,但基于扩散的视觉语言模型在仅用文本引导时仍难以精确控制生成图像中物体的形状、外观和位置。现有全局图像编辑模型依赖额外掩码或图像作为引导,常需重新训练模型;局部物体编辑模型虽可修改物体形状,却无法控制其位置。为此,我们提出无掩码、无重训练的物体级布局控制扩散模型(MFTF),可在不使用额外掩码或图像的情况下实现物体位置的精准控制。MFTF支持单物体与多物体的位置调整,如平移与旋转,并能同时进行布局控制与语义编辑。该模型采用源与目标扩散模型的并行去噪过程,从源模型的交叉注意力层动态生成注意力掩码,作用于自注意力层查询以隔离物体;这些查询根据布局控制参数调整后,重新注入目标模型的自注意力层,从而确保物体位置的精确控制。项目源代码见 https://github.com/syang-genai/MFTF。

原文摘要 · Abstract (English)

Text-to-image generation models have revolutionized content creation, but diffusion-based vision-language models still face challenges in precisely controlling the shape, appearance, and positional placement of objects in generated images using text guidance alone. Existing global image editing models rely on additional masks or images as guidance to achieve layout control, often requiring retraining of the model. While local object-editing models allow modifications to object shapes, they lack the capability to control object positions. To address these limitations, we propose the Mask-free Training-free Object-Level Layout Control Diffusion Model (MFTF), which provides precise control over object positions without requiring additional masks or images. The MFTF model supports both single-object and multi-object positional adjustments, such as translation and rotation, while enabling simultaneous layout control and object semantic editing. The MFTF model employs a parallel denoising process for both the source and target diffusion models. During this process, attention masks are dynamically generated from the cross-attention layers of the source diffusion model and applied to queries from the self-attention layers to isolate objects. These queries, generated in the source diffusion model, are then adjusted according to the layout control parameters and re-injected into the self-attention layers of the target diffusion model. This approach ensures accurate and precise positional control of objects. Project source code available at https://github.com/syang-genai/MFTF.

扩散模型图像生成布局控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。