用大模型和注意力图实现无需训练的精准图像实例操作
LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps
- 结合大模型与扩散U-Net注意力图定位目标实例
- 无需掩码或微调即可实现物体位置精确控制
- 适合需要灵活编辑图像内容的研究者使用
文本到图像生成技术已能根据文本提示生成逼真的图像，但对图像属性的精确控制仍具挑战，尤其在实例级别。现有方法虽可通过微调或附加信息实现控制，但灵活性与准确性受限。本文提出一种新流程，利用大语言模型（LLMs）、开放词汇检测器、扩散U-Net的交叉注意力图及中间激活值，实现实例级图像操作。该方法可识别提示中提及且存在于生成图像中的物体，实现精准修改而无需额外训练或输入掩码。通过引入交叉注意力图，确保操作后图像的一致性并控制物体位置。本方法无需微调或辅助信息（如掩码、边界框），即可完成高精度实例级操作。代码已开源：https://github.com/Palandr123/DiffusionU-NetLLM
原文摘要 · Abstract (English)
The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at https://github.com/Palandr123/DiffusionU-NetLLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。