无需训练即可实现扩散Transformer的精准区域提示生成
Training-free Regional Prompting for Diffusion Transformers
- 通过注意力机制操控实现无需训练的区域提示
- 在FLUX.1上实现复杂文本提示的细粒度图像生成
- 适合需要精确控制生成内容布局的研究者
扩散模型在文生图任务中表现出色，结合大型语言模型（如T5、Llama）后，其语义理解能力显著提升。然而，现有模型在处理长而复杂的文本提示时仍存在不足，尤其当提示包含多个对象及其丰富的属性与空间关系时。尽管已有诸多针对UNet架构（如SD1.5、SDXL）的区域提示方法，但针对近期扩散Transformer（DiT）架构（如SD3、FLUX.1）的实现仍为空白。本文提出并实现了基于注意力操作的FLUX.1区域提示方法，以无训练方式赋予DiT细粒度组合式文生图能力。代码已开源：https://github.com/antonioo-c/Regional-Prompting-FLUX。
原文摘要 · Abstract (English)
Diffusion models have demonstrated excellent capabilities in text-to-image generation. Their semantic understanding (i.e., prompt following) ability has also been greatly improved with large language models (e.g., T5, Llama). However, existing models cannot perfectly handle long and complex text prompts, especially when the text prompts contain various objects with numerous attributes and interrelated spatial relationships. While many regional prompting methods have been proposed for UNet-based models (SD1.5, SDXL), but there are still no implementations based on the recent Diffusion Transformer (DiT) architecture, such as SD3 and FLUX.1.In this report, we propose and implement regional prompting for FLUX.1 based on attention manipulation, which enables DiT with fined-grained compositional text-to-image generation capability in a training-free manner. Code is available at https://github.com/antonioo-c/Regional-Prompting-FLUX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。