用运动轨迹控制视频生成，让动作更自然真实。
Motion Prompting: Controlling Video Generation with Motion Trajectories
- 用稀疏或稠密运动轨迹作为输入，灵活控制视频动态。
- 可实现相机、物体运动控制及图像互动，生成符合物理规律的动画。
- 适合需要精准动作控制的视频创作、动画设计人员。
运动控制对生成富有表现力和吸引力的视频内容至关重要；然而，现有视频生成模型主要依赖文本提示进行控制，难以捕捉动态动作与时间结构的细微差别。为此，我们训练了一个基于时空稀疏或稠密运动轨迹的视频生成模型。与以往运动条件化方法不同，该灵活表示可编码任意数量的轨迹、特定对象或全局场景运动，以及时间稀疏的运动；因其灵活性，我们称之为运动提示（motion prompts）。用户可直接指定稀疏轨迹，我们还展示了如何将高层用户请求转化为详细、半稠密的运动提示，这一过程称为运动提示扩展。我们在多种应用中验证了该方法的通用性，包括相机与物体运动控制、‘与图像互动’、运动迁移和图像编辑。结果展现出真实物理行为等涌现特性，表明运动提示在探索视频模型及未来生成世界模型交互中的潜力。最后，我们进行了量化评估和人工评测，表现优异。视频结果见：https://motion-prompting.github.io/
原文摘要 · Abstract (English)
Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions. To this end, we train a video generation model conditioned on spatio-temporally sparse or dense motion trajectories. In contrast to prior motion conditioning work, this flexible representation can encode any number of trajectories, object-specific or global scene motion, and temporally sparse motion; due to its flexibility we refer to this conditioning as motion prompts. While users may directly specify sparse trajectories, we also show how to translate high-level user requests into detailed, semi-dense motion prompts, a process we term motion prompt expansion. We demonstrate the versatility of our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing. Our results showcase emergent behaviors, such as realistic physics, suggesting the potential of motion prompts for probing video models and interacting with future generative world models. Finally, we evaluate quantitatively, conduct a human study, and demonstrate strong performance. Video results are available on our webpage: https://motion-prompting.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。