arXiv:2608.14790cs.CV2026-08被引 1

用图像编辑模型直接操作视频隐空间,实现低成本指令式视频编辑。

Instruction-Based Video Editing by Repurposing an Image Editing Model

论文配图:Instruction-Based Video Editing by Repurposing an Image Editing Model
图 1 · 摘自论文原文
  • 将视频帧隐表示为大图像块,复用图像编辑模型的编码机制。
  • 零样本即可完成可识别的视频编辑,微调后效果接近高质量生成。
  • 适合需要快速实现视频编辑且资源有限的研究者或开发者。

指令式视频编辑通常依赖预训练视频生成模型,需高昂成本进行条件调整。本文探索新路径:利用强指令式图像编辑模型直接在视频-VAE隐空间上操作。基于Qwen-Image-Edit,将Wan 2.1视频VAE的隐帧排列为一张大型虚拟图像,复用图像编辑器的位置编码,并通过轻量级输入/输出投影桥接隐空间,使初始状态下的静态视频被精确嵌入为模型已理解的图像形式。整个系统在公开的Ditto-1M编辑三元组上微调,辅以少量Wan 2.2去噪步骤作为可选时序增强。设计动机来自一系列零训练观察:原图像编辑器能处理拼贴式视频;对单个联合编码或逐帧编码拼接的隐令牌无感;甚至零样本即可对真实视频隐空间进行清晰可辨的编辑,仅需微调填补保真度差距。结果表明,尽管视频隐空间训练投入巨大,其逐帧表示仍足够接近图像域,使成熟图像编辑先验仅需最小适配即可迁移。项目页:https://yunpeng1998.github.io/Qwen-Video-Edit-Page 代码:https://github.com/yunpeng1998/Qwen-Video-Edit 模型:https://huggingface.co/yunpeng1998/Qwen-Video-Edit

原文摘要 · Abstract (English)

Instruction-based video editing is commonly built on video-pretrained generative backbones: a video diffusion transformer is adapted, at considerable cost, to condition on a source video and an editing instruction. In this report we explore a different route and show that a strong instruction-based image editing model can edit videos by operating directly on video-VAE latents. Starting from Qwen-Image-Edit, we arrange the latent frames of a Wan~2.1 video VAE as tiles of one large virtual image, reuse the editor's image positional encoding for every tile, and bridge the two latent spaces with a pair of lightweight input/output projections warm-started from the editor's own patchify and unpatchify layers, so that at initialization a (static) video is embedded exactly as an image the model already understands. The whole system is then fine-tuned on the public Ditto-1M editing triplets, and a few denoising steps of Wan~2.2 serve as an optional temporal enhancer. We motivate the design with a chain of zero-training observations: the stock image editor already edits a video presented as a contact sheet; it is indifferent to whether the sheet's tokens come from one joint encode or from per-frame encodes stitched in latent space; and it even edits genuine video latents zero-shot to a clearly recognizable degree, leaving fine-tuning only a fidelity gap to close. Our results suggest that, despite the large investment in training video latent spaces, per-frame video latents remain close enough to the image domain that mature image editing priors transfer with minimal adaptation. Project Page: https://yunpeng1998.github.io/Qwen-Video-Edit-Page Code: https://github.com/yunpeng1998/Qwen-Video-Edit Model: https://huggingface.co/yunpeng1998/Qwen-Video-Edit

视频编辑图像模型隐空间指令控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。