RealCam实现交互式实时视频视角生成,支持动态摄像机控制。
RealCam: Real-Time Novel-View Video Generation with Interactive Camera Control

- 采用跨帧上下文学习框架,突破传统序列依赖限制。
- 推理速度比现有方法快数个数量级,支持实时流式生成。
- 适合影视制作与直播场景,对动态镜头控制友好。
相机控制的视频到视频生成可从单目视频动态合成新视角,具有巨大潜力于交互式电影制作与直播。然而,现有隐式合成方法依赖非因果、全序列处理及刚性前缀式时间拼接,导致双向注意力带来极高计算延迟、二次复杂度增长,并不兼容实时流或可变长度输入。为此,我们提出 exttt{RealCam},一种用于交互式实时相机控制视频生成的自回归框架。首先设计基于 extbf{跨帧上下文学习} 的高保真教师模型,通过交错源帧与目标帧形成同步上下文对,实现长度无关泛化并自然支持因果适应,打破刚性前缀瓶颈。随后通过带分布匹配的自强迫蒸馏,将教师模型压缩为几步因果学生模型,实现实时流式合成。此外,为缓解闭环轨迹中的严重循环不一致问题,提出 extbf{环闭数据增强 (LoopAug)},从多视角数据集中合成全局一致的循环序列。大量实验表明, exttt{RealCam} 在视觉保真度与时间一致性上达到当前最优,同时实现真正交互式相机控制,推理速度比现有范式快数个数量级。
原文摘要 · Abstract (English)
Camera-controlled video-to-video (V2V) generation enables dynamic viewpoint synthesis from monocular footage, holding immense potential for interactive filmmaking and live broadcasting. However, existing implicit synthesis methods fundamentally rely on non-causal, full-sequence processing and rigid prefix-style temporal concatenation. This architectural paradigm mandates bidirectional attention, resulting in prohibitive computational latency, quadratic complexity scaling, and inherent incompatibility with real-time streaming or variable-length inputs. To overcome these limitations, we introduce \texttt{RealCam}, a novel autoregressive framework for interactive, real-time camera-controlled V2V generation. We first design a high-fidelity teacher model grounded in a \textbf{Cross-frame In-context Learning} paradigm. By interleaving source and target frames into synchronized contextual pairs, our design inherently enables length-agnostic generalization and naturally facilitates causal adaptation, breaking the rigid prefix bottleneck. We then distill this teacher into a few-step causal student via Self-Forcing with Distribution Matching Distillation, enabling efficient, on-the-fly streaming synthesis. Furthermore, to mitigate severe loop inconsistency in closed-loop trajectories, we propose \textbf{Loop-Closed Data Augmentation (LoopAug)}, a novel paradigm that synthesizes globally consistent loop sequences from existing multiview datasets. Extensive experiments demonstrate that \texttt{RealCam} achieves state-of-the-art visual fidelity and temporal consistency while enabling truly interactive camera control with orders-of-magnitude faster inference than existing paradigms. Our project page is at https://xyc-fly.github.io/RealCam/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。