arXiv:2410.12696cs.CV2024-10被引 7

无需掩码的语义驱动图像拖拽,让编辑更精准自然

AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editing

  • 通过超像素分割自动生成掩码,实现无须手动标注的点拖拽
  • 基于扩散模型优化潜在表示,支持重缩放、移动、延伸等多样操作
  • 语义引导的自适应步数控制,提升结果真实性和用户意图匹配度

近期出现的基于点的图像编辑方法(如 DragDiffusion、FreeDrag、DragNoise)在用户指令下可生成高精度、高质量的结果。然而,这些方法对语义信息利用不足,影响最终效果。本文提出一种新的无掩码点式图像编辑方法 AdaptiveDrag,提供更灵活的编辑方式,并生成更符合用户意图的图像。我们设计了一个基于超像素分割的自动掩码生成模块,提升易用性;利用预训练扩散模型优化潜在空间,将特征从控制点拖拽至目标点。为确保输入图像与拖拽过程的全面关联,提出语义驱动的优化策略,设计由点位置和超像素分割得到的语义区域监督的自适应步数。该优化过程使结果更真实准确。此外,为解决扩散模型生成一致性问题,我们在采样过程中引入创新的对应损失。仅需单张输入图像和一对控制-目标点,本方法在多个领域(动物、人脸、风景、服装)上均优于现有方法,能有效处理重缩放、移动、延伸等多样化拖拽指令。

原文摘要 · Abstract (English)

Recently, several point-based image editing methods (e.g., DragDiffusion, FreeDrag, DragNoise) have emerged, yielding precise and high-quality results based on user instructions. However, these methods often make insufficient use of semantic information, leading to less desirable results. In this paper, we proposed a novel mask-free point-based image editing method, AdaptiveDrag, which provides a more flexible editing approach and generates images that better align with user intent. Specifically, we design an auto mask generation module using super-pixel division for user-friendliness. Next, we leverage a pre-trained diffusion model to optimize the latent, enabling the dragging of features from handle points to target points. To ensure a comprehensive connection between the input image and the drag process, we have developed a semantic-driven optimization. We design adaptive steps that are supervised by the positions of the points and the semantic regions derived from super-pixel segmentation. This refined optimization process also leads to more realistic and accurate drag results. Furthermore, to address the limitations in the generative consistency of the diffusion model, we introduce an innovative corresponding loss during the sampling process. Building on these effective designs, our method delivers superior generation results using only the single input image and the handle-target point pairs. Extensive experiments have been conducted and demonstrate that the proposed method outperforms others in handling various drag instructions (e.g., resize, movement, extension) across different domains (e.g., animals, human face, land space, clothing).

图像编辑扩散模型点拖拽语义驱动

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。