让视频生成模型精准控制镜头视角,无需重新训练。
ReRoPE: Repurposing RoPE for Relative Camera Control

- 利用旋转位置编码的低频空位注入相对相机信息。
- 在图像到视频和视频到视频任务中实现高精度视角控制。
- 插件式设计,不改变模型结构,适合快速部署。
可控镜头视角的视频生成对交互式内容创作、游戏和仿真等应用至关重要。现有方法通常使用相对于固定参考帧(如第一帧)的相机位姿进行适配,但这类编码缺乏平移不变性,常导致泛化能力差和累积漂移。尽管任意视角对之间的相对相机位姿嵌入更具鲁棒性,但将其融入预训练视频扩散模型,且不带来高昂训练成本或架构修改仍具挑战。我们提出ReRoPE,一种即插即用框架,可在不损害生成能力的前提下,将相对相机信息引入预训练视频扩散模型。其核心思路是:现有模型中的旋转位置编码(RoPE)在低频分量上未充分利用其全频谱带宽。通过将相对相机位姿信息无缝注入这些未被充分利用的频段,ReRoPE实现了精确控制并保留了强大的预训练生成先验。我们在图像到视频(I2V)和视频到视频(V2V)任务上评估了该方法,在相机控制精度和视觉保真度方面均表现优异。结果表明,ReRoPE为实现高效、高保真可控视频生成提供了可行路径。更多结果见项目页面:https://sisyphe-lee.github.io/ReRoPE/
原文摘要 · Abstract (English)
Video generation with controllable camera viewpoints is essential for applications such as interactive content creation, gaming, and simulation. Existing methods typically adapt pre-trained video models using camera poses relative to a fixed reference, e.g., the first frame. However, these encodings lack shift-invariance, often leading to poor generalization and accumulated drift. While relative camera pose embeddings defined between arbitrary view pairs offer a more robust alternative, integrating them into pre-trained video diffusion models without prohibitive training costs or architectural changes remains challenging. We introduce ReRoPE, a plug-and-play framework that incorporates relative camera information into pre-trained video diffusion models without compromising their generation capability. Our approach is based on the insight that Rotary Positional Embeddings (RoPE) in existing models underutilize their full spectral bandwidth, particularly in the low-frequency components. By seamlessly injecting relative camera pose information into these underutilized bands, ReRoPE achieves precise control while preserving strong pre-trained generative priors. We evaluate our method on both image-to-video (I2V) and video-to-video (V2V) tasks in terms of camera control accuracy and visual fidelity. Our results demonstrate that ReRoPE offers a training-efficient path toward controllable, high-fidelity video generation. See project page for more results: https://sisyphe-lee.github.io/ReRoPE/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。