实现多主体视频定制，支持深度、掩码等多模态控制。
OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions
- 构建无标签数据流水线生成多主体训练对
- 提出扩散Transformer框架，支持多主体即时编辑
- 适合需要灵活控制视频主体的创作者
现有前馈式主体驱动视频定制方法因多主体训练数据难构建，多集中于单主体场景。如何利用深度、掩码、相机参数和文本提示等信号来控制和编辑定制视频中的主体仍少有研究。本文首次提出 VideoCus-Factory 数据构建流水线，从无标签原始视频中生成多主体定制训练对，包括 depth-to-video 与 mask-to-video 对。基于此数据，设计 Image-Video Transfer Mixed (IVTM) 训练策略，融合图像编辑数据实现主体指导性编辑。进一步提出 diffusion Transformer 框架 OmniVCus，包含 Lottery Embedding (LE) 与 Temporally Aligned Embedding (TAE) 两种嵌入机制：LE 通过训练主体激活更多帧嵌入以支持推理时更多主体；TAE 通过将控制信号与噪声令牌分配相同帧嵌入，促使生成过程提取时间对齐控制信号的引导。实验表明，该方法在定量与定性评估上均显著优于现有最优方法。视频演示见项目主页：https://caiyuanhao1998.github.io/project/OmniVCus/。代码、模型与数据已开源：https://github.com/caiyuanhao1998/Open-OmniVCus
原文摘要 · Abstract (English)
Existing feedforward subject-driven video customization methods mainly study single-subject scenarios due to the difficulty of constructing multi-subject training data pairs. Another challenging problem that how to use the signals such as depth, mask, camera, and text prompts to control and edit the subject in the customized video is still less explored. In this paper, we first propose a data construction pipeline, VideoCus-Factory, to produce training data pairs for multi-subject customization from raw videos without labels and control signals such as depth-to-video and mask-to-video pairs. Based on our constructed data, we develop an Image-Video Transfer Mixed (IVTM) training with image editing data to enable instructive editing for the subject in the customized video. Then we propose a diffusion Transformer framework, OmniVCus, with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned Embedding (TAE). LE enables inference with more subjects by using the training subjects to activate more frame embeddings. TAE encourages the generation process to extract guidance from temporally aligned control signals by assigning the same frame embeddings to the control and noise tokens. Experiments demonstrate that our method significantly surpasses state-of-the-art methods in both quantitative and qualitative evaluations. Video demos are at our project page: https://caiyuanhao1998.github.io/project/OmniVCus/. Our code, models, data are released at https://github.com/caiyuanhao1998/Open-OmniVCus
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。