用扩散模型精准重导单目视频相机轨迹，实现视角自由控制。
TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models
- 双流条件扩散模型，融合点云渲染与源视频，分离视角变换与内容生成。
- 在单目视频上实现高精度视角重定向，多视角与大规模数据集均表现优异。
- 创新双重重投影策略构建混合数据集，提升跨场景泛化能力，适合影视剪辑与VR应用。
我们提出TrajectoryCrafter，一种针对单目视频重导相机轨迹的新方法。通过将确定性视角变换与随机内容生成解耦，该方法可精确控制用户指定的相机轨迹。我们设计了一种新型双流条件视频扩散模型，同时以点云渲染和源视频作为条件，确保视角变换准确且4D内容连贯。不同于依赖稀缺多视角视频的方法，我们采用创新的双重重投影策略，融合网络规模的单目视频与静态多视角数据集，构建混合训练数据集，显著增强在多样化场景下的鲁棒泛化能力。在多视角及大规模单目视频上的广泛评估表明，本方法性能优越。
原文摘要 · Abstract (English)
We present TrajectoryCrafter, a novel approach to redirect camera trajectories for monocular videos. By disentangling deterministic view transformations from stochastic content generation, our method achieves precise control over user-specified camera trajectories. We propose a novel dual-stream conditional video diffusion model that concurrently integrates point cloud renders and source videos as conditions, ensuring accurate view transformations and coherent 4D content generation. Instead of leveraging scarce multi-view videos, we curate a hybrid training dataset combining web-scale monocular videos with static multi-view datasets, by our innovative double-reprojection strategy, significantly fostering robust generalization across diverse scenes. Extensive evaluations on multi-view and large-scale monocular videos demonstrate the superior performance of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。