利用视频扩散模型自身能力,零样本生成定制化视频。
VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models
- 直接用视频扩散模型提取参考图像特征,无需额外模型。
- 通过双向注意力机制注入特征,保持主体外观一致性。
- 在人物和物体生成上均实现高质量结果,适合创意生成场景。
零样本定制化视频生成因其广泛应用潜力受到广泛关注。现有方法依赖额外模型提取并注入参考主体特征,认为视频扩散模型(VDM)自身不足以完成该任务。然而,这些方法常因特征提取与注入技术不佳导致主体外观不一致。本文揭示VDM具备内在的特征提取与注入能力。不同于以往的启发式方法,我们提出新框架,利用VDM的内在力量实现高质量零样本定制化视频生成。具体而言,特征提取方面,直接将参考图像输入VDM,利用其固有特征提取过程,不仅获得细粒度特征,还与VDM预训练知识高度对齐。特征注入方面,设计了一种基于空间自注意力的双向交互机制,使主体特征与生成内容协同优化,显著提升主体保真度的同时保持视频多样性。在定制化人物与物体视频生成上的实验验证了该框架的有效性。
原文摘要 · Abstract (English)
Zero-shot customized video generation has gained significant attention due to its substantial application potential. Existing methods rely on additional models to extract and inject reference subject features, assuming that the Video Diffusion Model (VDM) alone is insufficient for zero-shot customized video generation. However, these methods often struggle to maintain consistent subject appearance due to suboptimal feature extraction and injection techniques. In this paper, we reveal that VDM inherently possesses the force to extract and inject subject features. Departing from previous heuristic approaches, we introduce a novel framework that leverages VDM's inherent force to enable high-quality zero-shot customized video generation. Specifically, for feature extraction, we directly input reference images into VDM and use its intrinsic feature extraction process, which not only provides fine-grained features but also significantly aligns with VDM's pre-trained knowledge. For feature injection, we devise an innovative bidirectional interaction between subject features and generated content through spatial self-attention within VDM, ensuring that VDM has better subject fidelity while maintaining the diversity of the generated video. Experiments on both customized human and object video generation validate the effectiveness of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。