arXiv:2603.15478cs.CV2026-03被引 1

仅用2D图像即可实现视频生成与编辑,无需视频训练数据。

ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer

  • 通过解耦空间与3D注意力结构,实现高效视频编辑
  • 仅需少量2D图像训练,即达高质量可控视频生成
  • 支持多种条件信号,适合视频编辑和生成研究者

扩散Transformer(DiTs)在图像与视频生成中展现出卓越的可扩展性与质量,推动其向可控生成与编辑任务延伸。然而,相较于图像模型,视频控制与编辑进展受限,主要因配对视频数据稀缺及视频扩散模型训练成本高昂。本文提出无需视频训练数据的视频自由调优框架ViFeEdit,仅依赖2D图像即可实现多样化的视频生成与编辑。核心在于架构重参数化,将空间独立性与全3D注意力解耦,使视觉保真度高且时间一致性好,额外参数极少。该设计采用双路径流水线,对噪声调度使用独立的时间步嵌入,具备强条件信号适应性。大量实验表明,仅在2D图像上进行极小规模训练,即可实现优异的可控视频生成与编辑效果。代码已公开于https://github.com/Lexie-YU/ViFeEdit。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) have demonstrated remarkable scalability and quality in image and video generation, prompting growing interest in extending them to controllable generation and editing tasks. However, compared to the image counterparts, progress in video control and editing remains limited, mainly due to the scarcity of paired video data and the high computational cost of training video diffusion models. To address this issue, in this paper, we propose a video-free tuning framework termed ViFeEdit for video diffusion transformers. Without requiring any forms of video training data, ViFeEdit achieves versatile video generation and editing, adapted solely with 2D images. At the core of our approach is an architectural reparameterization that decouples spatial independence from the full 3D attention in modern video diffusion transformers, which enables visually faithful editing while maintaining temporal consistency with only minimal additional parameters. Moreover, this design operates in a dual-path pipeline with separate timestep embeddings for noise scheduling, exhibiting strong adaptability to diverse conditioning signals. Extensive experiments demonstrate that our method delivers promising results of controllable video generation and editing with only minimal training on 2D image data. Codes are available https://github.com/Lexie-YU/ViFeEdit.

视频生成扩散模型图像编辑

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。