无需调参,一键将视频特效迁移至新视频或图像。
Tuning-free Visual Effect Transfer across Videos
- 以参考视频为条件,直接生成带动态效果的输出视频。
- 在定量和人类偏好测试中均优于仅用文本提示的方法。
- 适合需要快速迁移复杂动态特效的创作者与开发者。
我们提出RefVFX,一种无需调参的框架,可将复杂的时序特效从参考视频无须额外条件地迁移到目标视频或图像中。现有方法在基于提示或关键帧编辑方面表现良好,但在处理动态光照变化或角色变形等难以用文字描述的动态时序特效时表现不佳。该任务挑战在于需将新时序动态与输入视频的原有运动和外观融合。为此,我们构建了一个大规模三元组数据集,每个三元组包含参考效果视频、输入图像或视频及对应的效果迁移输出视频。其中视频到视频的三元组需通过可扩展的自动化流水线生成,确保保留输入动作结构的同时实现固定可重复的效果转换。我们还利用LoRA适配器生成图像到视频的三元组,并通过程序化组合生成代码驱动的时序特效进行数据增强。基于此数据集,我们使用最新的文本到视频骨干网络训练参考条件模型。实验表明,RefVFX能生成视觉一致且时间连贯的编辑结果,对未见特效类别具有泛化能力,在定量指标和人类偏好上均优于纯提示基线方法。
原文摘要 · Abstract (English)
We present RefVFX, a new framework that transfers complex temporal effects from a reference video onto a target video or image in a feed-forward manner. While existing methods excel at prompt-based or keyframe-conditioned editing, they struggle with dynamic temporal effects such as dynamic lighting changes or character transformations, which are difficult to describe via text or static conditions. Transferring a video effect is challenging, as the model must integrate the new temporal dynamics with the input video's existing motion and appearance. % To address this, we introduce a large-scale dataset of triplets, where each triplet consists of a reference effect video, an input image or video, and a corresponding output video depicting the transferred effect. Creating this data is non-trivial, especially the video-to-video effect triplets, which do not exist naturally. To generate these, we propose a scalable automated pipeline that creates high-quality paired videos designed to preserve the input's motion and structure while transforming it based on some fixed, repeatable effect. We then augment this data with image-to-video effects derived from LoRA adapters and code-based temporal effects generated through programmatic composition. Building on our new dataset, we train our reference-conditioned model using recent text-to-video backbones. Experimental results demonstrate that RefVFX produces visually consistent and temporally coherent edits, generalizes across unseen effect categories, and outperforms prompt-only baselines in both quantitative metrics and human preference. See our website at https://snap-research.github.io/RefVFX/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。