用文字和图片生成可控制的动态特效,让普通人也能做电影级视觉效果。
VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer
- 将特效生成视为图像动画,结合文本与参考图实现可控生成。
- 在15类特效上达到顶尖表现,时空控制精度显著优于现有方法。
- 适合影视创作者、设计师快速生成高质量动态特效,无需专业技能。
影视中的魔法与幻觉是创作魅力的核心,而视觉特效(VFX)正是其技术支柱。尽管生成式AI在图像和视频合成方面取得进展,但可控的VFX生成仍较少被研究。本文提出一种新的动画特效生成范式:以图像动画为基础,通过用户友好的文本描述和静态参考图生成动态效果。主要贡献包括:(i) 开源首个高质量的VFX视频数据集Open-VFX,涵盖15种特效类别,包含文本描述、实例分割掩码(用于空间控制)及起止时间戳(用于时间控制);(ii) VFX Creator框架,基于视频扩散变压器,引入空间与时间可控的LoRA适配器,仅需少量训练视频。通过即插即用的掩码控制模块实现实例级空间操控,结合嵌入扩散过程的分词化起止运动时间戳与文本编码器,实现对特效时机与节奏的精确控制。在Open-VFX测试集上的大量实验表明,该系统在真实感与动态性上均达领先水平,且具备优异的泛化能力。此外,我们设计了专用指标评估时间控制精度。本工作融合传统VFX技术与生成方法,为高效、高质量视频特效生成开辟新路径,使高级特效更易获取。
原文摘要 · Abstract (English)
Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences. While recent advances in generative artificial intelligence have driven progress in generic image and video synthesis, the domain of controllable VFX generation remains relatively underexplored. In this work, we propose a novel paradigm for animated VFX generation as image animation, where dynamic effects are generated from user-friendly textual descriptions and static reference images. Our work makes two primary contributions: (i) Open-VFX, the first high-quality VFX video dataset spanning 15 diverse effect categories, annotated with textual descriptions, instance segmentation masks for spatial conditioning, and start-end timestamps for temporal control. (ii) VFX Creator, a simple yet effective controllable VFX generation framework based on a Video Diffusion Transformer. The model incorporates a spatial and temporal controllable LoRA adapter, requiring minimal training videos. Specifically, a plug-and-play mask control module enables instance-level spatial manipulation, while tokenized start-end motion timestamps embedded in the diffusion process, alongside the text encoder, allow precise temporal control over effect timing and pace. Extensive experiments on the Open-VFX test set demonstrate the superiority of the proposed system in generating realistic and dynamic effects, achieving state-of-the-art performance and generalization ability in both spatial and temporal controllability. Furthermore, we introduce a specialized metric to evaluate the precision of temporal control. By bridging traditional VFX techniques with generative approaches, VFX Creator unlocks new possibilities for efficient and high-quality video effect generation, making advanced VFX accessible to a broader audience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。