统一编码相机参数,让视频生成更精准控制视角。
Unified Camera Positional Encoding for Controlled Video Generation
- 用光线编码统一表示相机位姿、内参和畸变,保持几何一致性。
- 在文本到视频生成中实现精确的初始视角控制,视觉质量领先。
- 仅增加不到1%参数,适合集成到现有扩散模型中,适合多视图任务。
Transformer已成为自动驾驶与具身智能中3D感知、视频生成及世界模型的通用骨干,理解相机几何对将视觉观测锚定于三维空间至关重要。然而,现有相机编码方法常依赖简化的针孔假设,难以泛化至真实世界相机的多样内参与镜头畸变。本文提出相对射线编码(Relative Ray Encoding),一种统一完整相机信息的几何一致表示,涵盖6-自由度位姿、内参及镜头畸变。为评估其在多样化可控性需求下的表现,采用相机控制的文本到视频生成作为测试任务。在此设定下,进一步发现俯仰角与翻滚角是实现绝对朝向编码的有效组件,可实现初始相机朝向的完全控制。上述设计共同构成UCPE(统一相机位置编码),通过轻量级空间注意力适配器集成至预训练视频扩散变压器,增加参数不足1%,却实现最先进的相机可控性与视觉保真度。为支持系统化训练与评估,构建了覆盖广泛相机运动与镜头类型的大型视频数据集。大量实验证明UCPE在相机可控视频生成中的有效性,并凸显其作为未来多视图、视频与3D任务中通用相机表示的潜力。代码将公开于https://github.com/chengzhag/UCPE。
原文摘要 · Abstract (English)
Transformers have emerged as a universal backbone across 3D perception, video generation, and world models for autonomous driving and embodied AI, where understanding camera geometry is essential for grounding visual observations in three-dimensional space. However, existing camera encoding methods often rely on simplified pinhole assumptions, restricting generalization across the diverse intrinsics and lens distortions in real-world cameras. We introduce Relative Ray Encoding, a geometry-consistent representation that unifies complete camera information, including 6-DoF poses, intrinsics, and lens distortions. To evaluate its capability under diverse controllability demands, we adopt camera-controlled text-to-video generation as a testbed task. Within this setting, we further identify pitch and roll as two components effective for Absolute Orientation Encoding, enabling full control over the initial camera orientation. Together, these designs form UCPE (Unified Camera Positional Encoding), which integrates into a pretrained video Diffusion Transformer through a lightweight spatial attention adapter, adding less than 1% trainable parameters while achieving state-of-the-art camera controllability and visual fidelity. To facilitate systematic training and evaluation, we construct a large video dataset covering a wide range of camera motions and lens types. Extensive experiments validate the effectiveness of UCPE in camera-controllable video generation and highlight its potential as a general camera representation for Transformers across future multi-view, video, and 3D tasks. Code will be available at https://github.com/chengzhag/UCPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。