用视频扩散模型提升交互式图像编辑,更高效且效果自然。
FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors
- 将编辑任务转为图到视频生成,借用视频扩散先验
- 仅需少量训练数据,实现无缝连贯的图像修改
- 支持真实与虚构场景的通用编辑,适合内容创作
交互式图像编辑允许用户通过绘画、点击、拖拽等视觉操作修改图像。现有方法依赖视频构建监督信号,因其能捕捉物体在物理互动下的变化。但这些模型通常基于文生图扩散模型,需大量训练样本及额外参考编码器以学习真实世界动态和视觉一致性。本文将该任务重新定义为图到视频生成问题,直接继承强大的视频扩散先验,降低训练成本并保证时序一致性。我们提出 FramePainter,基于 Stable Video Diffusion 构建,仅使用轻量级稀疏控制编码器注入编辑信号。针对时序注意力在大运动帧间处理能力不足的问题,进一步引入匹配注意力,扩大感受野并强化编辑图与源图令牌间的密集对应关系。实验表明,FramePainter 在多种编辑信号下显著优于先前最优方法,仅用极少训练数据即可实现高度连贯自然的图像编辑,例如自动调整杯子的倒影。此外,其在真实视频未覆盖的场景中也表现出卓越泛化能力,如将小丑鱼变为鲨鱼形态。代码将于 https://github.com/YBYBZhang/FramePainter 公开。
原文摘要 · Abstract (English)
Interactive image editing allows users to modify images through visual interaction operations such as drawing, clicking, and dragging. Existing methods construct such supervision signals from videos, as they capture how objects change with various physical interactions. However, these models are usually built upon text-to-image diffusion models, so necessitate (i) massive training samples and (ii) an additional reference encoder to learn real-world dynamics and visual consistency. In this paper, we reformulate this task as an image-to-video generation problem, so that inherit powerful video diffusion priors to reduce training costs and ensure temporal consistency. Specifically, we introduce FramePainter as an efficient instantiation of this formulation. Initialized with Stable Video Diffusion, it only uses a lightweight sparse control encoder to inject editing signals. Considering the limitations of temporal attention in handling large motion between two frames, we further propose matching attention to enlarge the receptive field while encouraging dense correspondence between edited and source image tokens. We highlight the effectiveness and efficiency of FramePainter across various of editing signals: it domainantly outperforms previous state-of-the-art methods with far less training data, achieving highly seamless and coherent editing of images, \eg, automatically adjust the reflection of the cup. Moreover, FramePainter also exhibits exceptional generalization in scenarios not present in real-world videos, \eg, transform the clownfish into shark-like shape. Our code will be available at https://github.com/YBYBZhang/FramePainter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。