提升修复流模型的反演精度,实现无需训练的高质量图像视频编辑
Taming Rectified Flow for Inversion and Editing
- 通过高阶泰勒展开精确求解修复流微分方程,减少反演误差
- 反演结果保留结构信息,使编辑后图像视频质量显著提升
- 兼容所有预训练修复流模型,适用于图像与视频编辑任务
基于修复流的扩散变换器(如FLUX和OpenSora)在图像与视频生成领域表现卓越。然而,这些模型常因反演不准确而限制下游任务(如图像视频编辑)效果。为此,我们提出RF-Solver——一种无需训练的采样器,通过推导修复流微分方程的精确表达式,并利用高阶泰勒展开估计其非线性项,显著提升每步的微分方程求解精度。在此基础上,我们进一步提出RF-Edit,一种通用的特征共享型编辑框架。该框架将反演过程中的自注意力特征引入编辑流程,有效保留源图像或视频的结构信息,同时实现高质量编辑。方法兼容任意预训练的修复流模型,无需额外训练或优化。跨图像与视频模态的生成、反演与编辑实验验证了方法的优越性与通用性。代码已开源:https://github.com/wangjiangshan0725/RF-Solver-Edit。
原文摘要 · Abstract (English)
Rectified-flow-based diffusion transformers like FLUX and OpenSora have demonstrated outstanding performance in the field of image and video generation. Despite their robust generative capabilities, these models often struggle with inversion inaccuracies, which could further limit their effectiveness in downstream tasks such as image and video editing. To address this issue, we propose RF-Solver, a novel training-free sampler that effectively enhances inversion precision by mitigating the errors in the ODE-solving process of rectified flow. Specifically, we derive the exact formulation of the rectified flow ODE and apply the high-order Taylor expansion to estimate its nonlinear components, significantly enhancing the precision of ODE solutions at each timestep. Building upon RF-Solver, we further propose RF-Edit, a general feature-sharing-based framework for image and video editing. By incorporating self-attention features from the inversion process into the editing process, RF-Edit effectively preserves the structural information of the source image or video while achieving high-quality editing results. Our approach is compatible with any pre-trained rectified-flow-based models for image and video tasks, requiring no additional training or optimization. Extensive experiments across generation, inversion, and editing tasks in both image and video modalities demonstrate the superiority and versatility of our method. The source code is available at https://github.com/wangjiangshan0725/RF-Solver-Edit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。