无需微调，用预训练模型自引导实现图像到视频的精准运动控制。
SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
- 利用预训练模型自身知识实现零样本运动控制。
- 在视觉质量和运动保真度上接近有监督模型表现。
- 适合需要快速生成可控视频且无标注数据的场景。
图像到视频生成方法已达到出色的逼真度。然而，调整生成视频中物体运动或摄像机移动等特定元素，通常需反复试错，例如通过不同随机种子重新生成视频。近期技术通过微调预训练模型来响应边界框或点轨迹等条件信号来解决此问题，但该过程计算成本高，且需带有物体运动标注的数据集，获取困难。本文提出SG-I2V框架，实现无需微调、仅依赖预训练图像到视频扩散模型内部知识的自引导可控生成。该零样本方法在视觉质量与运动保真度上优于无监督基线，并显著缩小了与有监督模型的差距。更多细节与视频结果见项目页：https://kmcode1.github.io/Projects/SG-I2V。
原文摘要 · Abstract (English)
Methods for image-to-video generation have achieved impressive, photo-realistic quality. However, adjusting specific elements in generated videos, such as object motion or camera movement, is often a tedious process of trial and error, e.g., involving re-generating videos with different random seeds. Recent techniques address this issue by fine-tuning a pre-trained model to follow conditioning signals, such as bounding boxes or point trajectories. Yet, this fine-tuning procedure can be computationally expensive, and it requires datasets with annotated object motion, which can be difficult to procure. In this work, we introduce SG-I2V, a framework for controllable image-to-video generation that is self-guided$\unicode{x2013}$offering zero-shot control by relying solely on the knowledge present in a pre-trained image-to-video diffusion model without the need for fine-tuning or external knowledge. Our zero-shot method outperforms unsupervised baselines while significantly narrowing down the performance gap with supervised models in terms of visual quality and motion fidelity. Additional details and video results are available on our project page: https://kmcode1.github.io/Projects/SG-I2V
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。