用合成视频教会扩散模型学习非常规运镜，无需真实拍摄数据。
VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos
- 通过合成视频训练扩散模型，分离镜头运动与画面外观干扰。
- 仅用低多边形3D场景和简单几何体就能生成复杂精准的运镜效果。
- 适合需要艺术化、非现实运镜的视频生成研究者使用。
尽管近期文本到视频生成模型在遵循外部相机控制（如文本描述或轨迹）方面愈发强大，但仍难以泛化到非常规相机运动，而这对于创作真正原创且具有艺术性的视频至关重要。挑战在于难以获取足够数量包含目标罕见运镜的真实训练视频。为此，我们提出VividCam，一种使扩散模型能够从合成视频中学习复杂相机运动的训练范式，从而摆脱对真实训练视频的依赖。VividCam结合多种解耦策略，将相机运动学习与合成外观伪影隔离，确保更稳健的运动表征并缓解领域偏移问题。我们证明，该设计仅需相对简单的合成数据即可生成广泛而精确控制的复杂相机运动。值得注意的是，这些合成数据通常由低多边形3D场景中的基本几何体构成，可由Unity等引擎高效渲染。相关视频结果请见 https://wuqiuche.github.io/VividCamDemoPage/。
原文摘要 · Abstract (English)
Although recent text-to-video generative models are getting more capable of following external camera controls, imposed by either text descriptions or camera trajectories, they still struggle to generalize to unconventional camera motions, which is crucial in creating truly original and artistic videos. The challenge lies in the difficulty of finding sufficient training videos with the intended uncommon camera motions. To address this challenge, we propose VividCam, a training paradigm that enables diffusion models to learn complex camera motions from synthetic videos, releasing the reliance on collecting realistic training videos. VividCam incorporates multiple disentanglement strategies that isolates camera motion learning from synthetic appearance artifacts, ensuring more robust motion representation and mitigating domain shift. We demonstrate that our design synthesizes a wide range of precisely controlled and complex camera motions using surprisingly simple synthetic data. Notably, this synthetic data often consists of basic geometries within a low-poly 3D scene and can be efficiently rendered by engines like Unity. Our video results can be found in https://wuqiuche.github.io/VividCamDemoPage/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。