arXiv:2604.09429cs.CVcs.AI2026-04中稿 · ICML被引 3

用统一模型同时学视频和相机轨迹,实现镜头控制的视频生成。

Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories

论文配图:Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories
图 1 · 摘自论文原文
  • 将相机表示为与视频同空间的射线像素(raxels),联合建模视频与轨迹。
  • 单个模型可完成轨迹预测、镜头控制生成、联合合成三任务。
  • 首次实现端到端相机控制视频生成,适合视频生成与视觉重建研究者。

从图像中恢复相机参数与从新视角渲染场景在计算机视觉与图形学中长期被视为独立任务。当图像覆盖稀疏或位姿模糊时,两者相互依赖,分离处理会失效。本文提出「Rays as Pixels」——一种视频扩散模型(VDM),学习视频与相机轨迹的联合分布。据我们所知,这是首个在单一框架内同时预测相机位姿并实现相机控制视频生成的模型。我们将每个相机表示为密集射线像素(raxels),一种与视频帧共享潜空间的像素对齐编码,并通过解耦自-交叉注意力机制联合去噪。一个训练好的模型可处理三项任务:从视频预测相机轨迹、沿预设轨迹生成视频、以及从输入图像联合合成视频与轨迹。我们在姿态估计与相机控制视频生成上进行评估,并引入闭环自一致性测试,验证模型预测位姿与其条件渲染结果的一致性。消融实验表明,将相机嵌入与视频相同的潜空间显著优于普吕克嵌入(Plücker embeddings)。

原文摘要 · Abstract (English)

Recovering camera parameters from images and rendering scenes from novel viewpoints have been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task depends on what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. To our knowledge, this is the first model to predict camera poses and do camera-controlled video generation within a single framework. We represent each camera as dense ray pixels (raxels), a pixel-aligned encoding that lives in the same latent space as video frames, and denoise the two jointly through a Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, generating video from input images along a pre-defined trajectory, and jointly synthesizing video and trajectory from input images. We evaluate on pose estimation and camera-controlled video generation, and introduce a closed-loop self-consistency test showing that the model's predicted poses and its renderings conditioned on those poses agree. Ablations against Plücker embeddings confirm that representing cameras in a shared latent space with video is subtantially more effective.

视频生成扩散模型相机位姿联合建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。