无需反演和训练,通过像素操作实现物体编辑一致性
PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation
- 直接在像素空间复制目标物体到新位置
- 16步内完成编辑,效果优于多数需50步的方法
- 适合追求高效一致图像编辑的开发者使用
近期研究探索扩散模型在一致物体编辑中的潜力,旨在修改物体位置、大小和构图等,同时保持物体与背景的一致性,不改变其纹理和属性。当前基于推理的方法常依赖DDIM反演,影响效率与编辑一致性;也有方法采用能量引导,迭代更新预测噪声,但可能导致潜在空间偏移和图像失真。本文提出PixelMan,一种无需反演和训练的方法,通过像素级操作与生成实现一致编辑:直接在像素空间将源物体复制至目标位置,并采用高效采样策略迭代融合该物体至目标区域,同时修复原位置;通过锚定生成过程至像素操作后的图像,并引入多种一致性保持优化技术,保障整体一致性。在基准数据集上的实验及大量视觉对比表明,仅需16次推理步骤,PixelMan即超越多个先进训练型与训练自由方法(通常需50步),在多类一致物体编辑任务中表现更优。
原文摘要 · Abstract (English)
Recent research explores the potential of Diffusion Models (DMs) for consistent object editing, which aims to modify object position, size, and composition, etc., while preserving the consistency of objects and background without changing their texture and attributes. Current inference-time methods often rely on DDIM inversion, which inherently compromises efficiency and the achievable consistency of edited images. Recent methods also utilize energy guidance which iteratively updates the predicted noise and can drive the latents away from the original image, resulting in distortions. In this paper, we propose PixelMan, an inversion-free and training-free method for achieving consistent object editing via Pixel Manipulation and generation, where we directly create a duplicate copy of the source object at target location in the pixel space, and introduce an efficient sampling approach to iteratively harmonize the manipulated object into the target location and inpaint its original location, while ensuring image consistency by anchoring the edited image to be generated to the pixel-manipulated image as well as by introducing various consistency-preserving optimization techniques during inference. Experimental evaluations based on benchmark datasets as well as extensive visual comparisons show that in as few as 16 inference steps, PixelMan outperforms a range of state-of-the-art training-based and training-free methods (usually requiring 50 steps) on multiple consistent object editing tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。