从单视角生成持续动态场景,保持3D运动一致性。
Voyaging into Perpetual Dynamic Scenes from a Single View
- 将动态场景生成转为带新内容的场景外绘问题。
- 利用射线信息增强2D像素,实现3D运动一致性生成。
- 支持文本控制,适用于虚拟现实与机器人导航。
从单视角生成持续动态场景是增强现实、虚拟现实和机器人领域的重要问题。由于动态场景随时间变化,关键挑战在于确保不同生成视角间保持底层3D运动的一致性。以往方法依赖多视角训练,但生成区域常在训练视图间插值,无法实现持续生成。为此,我们提出DynamicVoyager,将动态场景生成重构为包含新动态内容的场景外绘问题。由于2D外绘模型难以从单个2D视图生成3D一致的运动,我们通过引入3D射线信息(如射线到点云的距离)丰富2D像素,以促进3D运动一致性学习。具体而言,首先使用估计的视频深度将单视角视频映射为动态点云;随后从新视角渲染部分视频,并利用射线信息外绘缺失区域,生成具有3D一致性的运动。接着,用外绘后的视频更新点云,用于后续新视角的外绘。此外,可通过输入文本提示控制生成内容。实验表明,该模型可沿飞行相机路径生成具有持续一致运动的场景。项目页:https://tianfr.github.io/DynamicVoyager。
原文摘要 · Abstract (English)
The problem of generating a perpetual dynamic scene from a single view is an important problem with widespread applications in augmented and virtual reality, and robotics. However, since dynamic scenes regularly change over time, a key challenge is to ensure that different generated views be consistent with the underlying 3D motions. Prior work learns such consistency by training on multiple views, but the generated scene regions often interpolate between training views and fail to generate perpetual views. To address this issue, we propose DynamicVoyager, which reformulates dynamic scene generation as a scene outpainting problem with new dynamic content. As 2D outpainting models struggle at generating 3D consistent motions from a single 2D view, we enrich 2D pixels with information from their 3D rays that facilitates learning of 3D motion consistency. More specifically, we first map the single-view video input to a dynamic point cloud using the estimated video depths. We then render a partial video of the point cloud from a novel view and outpaint the missing regions using ray information (e.g., the distance from a ray to the point cloud) to generate 3D consistent motions. Next, we use the outpainted video to update the point cloud, which is used for outpainting the scene from future novel views. Moreover, we can control the generated content with the input text prompt. Experiments show that our model can generate perpetual scenes with consistent motions along fly-through cameras. Project page: https://tianfr.github.io/DynamicVoyager.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。