让视频生成模型统一控制边缘、分割、人体关键点等细节
Enabling Versatile Controls for Video Diffusion Models
- 用通用条件模块统一编码多种控制信号，不修改生成器
- 在多个数据集上提升生成质量和可控性，人类评估更优
- 适合需要精细控制视频生成的创作者和研究者
尽管文本到视频生成已取得显著进展，但对细粒度时空属性的精确灵活控制仍是未解决的关键挑战。为此，我们提出VCtrl（又称PP-VCtrl）框架，通过通用条件模块统一集成用户指定的多种控制信号（如Canny边缘、语义分割掩码、人体关键点），实现对预训练视频扩散模型的细粒度控制。该方法无需修改底层生成器，采用统一的控制信号编码流程与稀疏残差连接机制，高效融合控制表示。大量实验和人工评估表明，VCtrl显著提升生成质量与可控性。代码与预训练模型已开源，基于PaddlePaddle框架，地址：http://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl。
原文摘要 · Abstract (English)
Despite substantial progress in text-to-video generation, achieving precise and flexible control over fine-grained spatiotemporal attributes remains a significant unresolved challenge in video generation research. To address these limitations, we introduce VCtrl (also termed PP-VCtrl), a novel framework designed to enable fine-grained control over pre-trained video diffusion models in a unified manner. VCtrl integrates diverse user-specified control signals-such as Canny edges, segmentation masks, and human keypoints-into pretrained video diffusion models via a generalizable conditional module capable of uniformly encoding multiple types of auxiliary signals without modifying the underlying generator. Additionally, we design a unified control signal encoding pipeline and a sparse residual connection mechanism to efficiently incorporate control representations. Comprehensive experiments and human evaluations demonstrate that VCtrl effectively enhances controllability and generation quality. The source code and pre-trained models are publicly available and implemented using the PaddlePaddle framework at http://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。