arXiv:2502.10059cs.CV2025-02ICCV被引 59

让真实图片生成视频时能精准控制相机运动,无需懂深度和尺度。

RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control

论文配图:RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control
图 1 · 摘自论文原文
  • 用单目深度估计重建3D场景,自动校准相机参数尺度。
  • 用户可直接在3D场景中拖拽绘制镜头轨迹,操作直观。
  • 支持循环视频生成与帧间插值,适合影视制作等场景。

基于相机轨迹的图像到视频生成方法相比文本引导更具精度和复杂相机控制能力,但实际应用中用户难以对任意真实图像提供精确相机参数,尤其缺乏深度和场景尺度信息。为此,我们提出 RealCam-I2V,一种基于扩散模型的视频生成框架,通过预处理阶段的单目度量深度估计实现3D场景重建。训练时,重建的3D场景使相机参数从相对尺度转换为度量尺度,确保跨不同真实图像的兼容性与尺度一致性。推理时,RealCam-I2V 提供直观界面,用户可在3D场景中通过拖拽精确绘制相机轨迹。为进一步提升相机控制精度与场景一致性,提出场景约束噪声调制机制,既能调控高层噪声,又能在低噪声阶段保持动态连贯的视频生成。RealCam-I2V 在 RealEstate10K 数据集及域外图像上显著提升可控性与视频质量,并支持相机控制的循环视频生成与生成式帧插值。项目主页:https://zgctroy.github.io/RealCam-I2V。

原文摘要 · Abstract (English)

Recent advancements in camera-trajectory-guided image-to-video generation offer higher precision and better support for complex camera control compared to text-based approaches. However, they also introduce significant usability challenges, as users often struggle to provide precise camera parameters when working with arbitrary real-world images without knowledge of their depth nor scene scale. To address these real-world application issues, we propose RealCam-I2V, a novel diffusion-based video generation framework that integrates monocular metric depth estimation to establish 3D scene reconstruction in a preprocessing step. During training, the reconstructed 3D scene enables scaling camera parameters from relative to metric scales, ensuring compatibility and scale consistency across diverse real-world images. In inference, RealCam-I2V offers an intuitive interface where users can precisely draw camera trajectories by dragging within the 3D scene. To further enhance precise camera control and scene consistency, we propose scene-constrained noise shaping, which shapes high-level noise and also allows the framework to maintain dynamic and coherent video generation in lower noise stages. RealCam-I2V achieves significant improvements in controllability and video quality on the RealEstate10K and out-of-domain images. We further enables applications like camera-controlled looping video generation and generative frame interpolation. Project page: https://zgctroy.github.io/RealCam-I2V.

图像转视频相机控制扩散模型3D重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。