arXiv:2410.11815cs.CV2024-10中稿 · SIGGRAPH被引 14

用大模型+文生图模型实现精准场景图编辑,改物不乱景。

SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing

  • 大模型解析图像生成结构化场景图,提取物体与关系
  • 微调扩散模型用优化标记和提示词表示每个物体
  • 大模型指导编辑,注意力调制实现精准增删改

场景图以节点和边的形式结构化表达图像中的物体及其相互关系,可作为自然的图像编辑接口,显著提升编辑精度与灵活性。本文提出一种融合大语言模型(LLM)与文本生成图像模型的场景图驱动图像编辑框架。该框架包含两个阶段:首先,利用基于LLM的场景解析器构建图像的场景图,捕捉关键物体、其关系以及细粒度属性(如物体掩码和描述);这些标注用于微调扩散模型,为每个物体生成优化的标记和详细描述提示词。其次,在编辑阶段,由LLM编辑控制器引导修改目标区域,通过注意力调制的扩散编辑器执行对象添加、删除、替换与调整,使用微调后的模型保持整体图像完整性。大量实验表明,该框架在编辑精度和场景美学上均显著优于现有方法。

原文摘要 · Abstract (English)

Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics.

图像编辑场景图大模型扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。