用单段视频生成新视角画面,保持动态一致性。
ReCamMaster: Camera-Controlled Generative Rendering from A Single Video

- 通过视频条件控制预训练文生视频模型生成新镜头。
- 在多摄像头数据集上训练,支持真实拍摄风格的镜头迁移。
- 适合视频修复、超分辨率与画外扩展等应用。
相机控制在文本或图像引导的视频生成中已受到广泛关注,但对给定视频的相机轨迹进行修改仍研究不足,尽管其在视频创作中至关重要。这具有挑战性,因为需同时维持多帧外观一致性和动态同步性。为此,我们提出ReCamMaster——一种基于单个视频的相机可控生成重渲染框架,可在新的相机轨迹下重现输入视频的动态场景。核心创新在于利用预训练文生视频模型的生成能力,通过一个简单而强大的视频条件机制实现控制,该能力在当前研究中常被忽视。为解决高质量训练数据稀缺问题,我们使用Unreal Engine 5构建了一个全面的多摄像机同步视频数据集,精心设计以符合真实拍摄特征,涵盖多样场景与运动方式,有助于模型泛化至真实视频。最后,通过精心设计的训练策略进一步提升对多样化输入的鲁棒性。大量实验表明,该方法显著优于现有最先进方法。本方法在视频稳定、超分辨率和外延补全方面也展现出良好应用前景。代码与数据集已公开:https://github.com/KwaiVGI/ReCamMaster。
原文摘要 · Abstract (English)
Camera control has been actively studied in text or image conditioned video generation tasks. However, altering camera trajectories of a given video remains under-explored, despite its importance in the field of video creation. It is non-trivial due to the extra constraints of maintaining multiple-frame appearance and dynamic synchronization. To address this, we present ReCamMaster, a camera-controlled generative video re-rendering framework that reproduces the dynamic scene of an input video at novel camera trajectories. The core innovation lies in harnessing the generative capabilities of pre-trained text-to-video models through a simple yet powerful video conditioning mechanism--its capability is often overlooked in current research. To overcome the scarcity of qualified training data, we construct a comprehensive multi-camera synchronized video dataset using Unreal Engine 5, which is carefully curated to follow real-world filming characteristics, covering diverse scenes and camera movements. It helps the model generalize to in-the-wild videos. Lastly, we further improve the robustness to diverse inputs through a meticulously designed training strategy. Extensive experiments show that our method substantially outperforms existing state-of-the-art approaches. Our method also finds promising applications in video stabilization, super-resolution, and outpainting. Our code and dataset are publicly available at: https://github.com/KwaiVGI/ReCamMaster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。