用大模型分步生成3D概念,再组合出可控的高质量视频。
Compositional 3D-aware Video Generation with LLM Director

- 分步生成3D概念,由大模型拆解提示并调用专家模型获取
- 通过多模态大模型生成物体运动轨迹粗略引导
- 结合2D扩散模型优化画面真实感,支持灵活控制每个元素
文本到视频生成已取得显著进展,但对视频中特定角色的运动、外观及视角移动等概念的精确控制仍面临挑战。本文提出一种新范式:先分别生成每个概念的3D表示,再结合大语言模型(LLM)与2D扩散模型的先验知识进行组合。具体流程包括:1)以LLM为导演,将复杂提示分解为多个子提示(如场景、物体、动作),并调用预训练专家模型获取对应3D表示;2)通过多模态LLM生成物体轨迹的尺度与坐标粗略引导;3)利用2D扩散模型先验和得分蒸馏采样(Score Distillation Sampling)使生成帧符合自然图像分布。大量实验表明,该方法可从文本生成高保真视频,实现多样运动与对各概念的灵活控制。
原文摘要 · Abstract (English)
Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement of viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt, our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video~(\textit{e.g.}, scene, objects, motions), then we let LLM to invoke pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates of trajectories for the objects. 3) To make the generated frames adhere to natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept. Project page: \url{https://aka.ms/c3v}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。