用扩散Transformer实现自动视频合成，可交互调整元素大小与轨迹。
GenCompositor: Generative Video Compositing with Diffusion Transformer
- 基于DiT设计新架构，支持动态元素自适应注入。
- 在61K数据集上实现高保真与一致性，优于现有方法。
- 适合影视制作、创意设计人员快速生成复杂视频场景。
视频合成将实拍画面组合成视频内容，是影视制作的关键技术。传统流程依赖人工与专家协作，耗时且成本高。为此，我们提出生成式视频合成，通过生成模型自动化完成。该任务旨在交互式地将前景视频的身份与运动信息适配到目标视频中，允许用户自定义添加元素的尺寸、运动轨迹等属性。我们设计了一种基于内在特性的新型扩散Transformer（DiT）流水线。为保持编辑前后背景一致性，采用轻量级DiT背景保留分支，结合掩码标记注入；为融合源视频动态元素，提出使用全自注意力的DiT融合模块，并引入简单有效的前景增强训练策略。此外，针对用户控制下背景与前景布局差异的问题，开发了名为扩展旋转位置编码（ERoPE）的新位置嵌入。最后，我们构建了包含61,000组视频的数据集VideoComp，涵盖完整动态元素与高质量目标视频。实验表明，本方法在保真度与一致性上显著优于现有方案。
原文摘要 · Abstract (English)
Video compositing combines live-action footage to create video production, serving as a crucial technique in video creation and film production. Traditional pipelines require intensive labor efforts and expert collaboration, resulting in lengthy production cycles and high manpower costs. To address this issue, we automate this process with generative models, called generative video compositing. This new task strives to adaptively inject identity and motion information of foreground video to the target video in an interactive manner, allowing users to customize the size, motion trajectory, and other attributes of the dynamic elements added in final video. Specifically, we designed a novel Diffusion Transformer (DiT) pipeline based on its intrinsic properties. To maintain consistency of the target video before and after editing, we revised a light-weight DiT-based background preservation branch with masked token injection. As to inherit dynamic elements from other sources, a DiT fusion block is proposed using full self-attention, along with a simple yet effective foreground augmentation for training. Besides, for fusing background and foreground videos with different layouts based on user control, we developed a novel position embedding, named Extended Rotary Position Embedding (ERoPE). Finally, we curated a dataset comprising 61K sets of videos for our new task, called VideoComp. This data includes complete dynamic elements and high-quality target videos. Experiments demonstrate that our method effectively realizes generative video compositing, outperforming existing possible solutions in fidelity and consistency. Project is available at https://gencompositor.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。