将单目视频转为立体3D视频,无需深度估计直接生成新视角。
Eye2Eye: A Simple Approach for Monocular-to-Stereo Video Synthesis
- 利用预训练视频模型的几何与材质先验,直接合成新视角。
- 在镜面、透明物体等复杂场景中表现优于传统分步方法。
- 适合影视制作、VR内容生成等需要快速3D化场景的应用。
沉浸式视觉体验的兴起推动了立体3D视频生成的研究。尽管视频合成技术进展显著,但3D视频数据稀缺仍带来挑战。本文提出一种简单方法,将文本到视频生成器转化为视频到立体视频生成器。给定输入视频,框架可自动生成偏移视角下的帧,实现逼真的3D效果。以往方法通常分多阶段:先估计视差或深度,再进行图像扭曲生成第二视角,最后修复遮挡区域。该方法在镜面或透明物体场景中因单层视差估计不足而失效,导致错位和伪影。本文工作绕过这些限制,直接合成新视角,不依赖外部几何模型,也无需显式解耦几何信息。通过利用预训练视频模型对几何、材质、光学和语义的先验知识,实现端到端生成。我们在包含多种材质和复杂构图的真实世界场景中验证了该方法的优势。视频展示见 https://video-eye2eye.github.io
原文摘要 · Abstract (English)
The rising popularity of immersive visual experiences has increased interest in stereoscopic 3D video generation. Despite significant advances in video synthesis, creating 3D videos remains challenging due to the relative scarcity of 3D video data. We propose a simple approach for transforming a text-to-video generator into a video-to-stereo generator. Given an input video, our framework automatically produces the video frames from a shifted viewpoint, enabling a compelling 3D effect. Prior and concurrent approaches for this task typically operate in multiple phases, first estimating video disparity or depth, then warping the video accordingly to produce a second view, and finally inpainting the disoccluded regions. This approach inherently fails when the scene involves specular surfaces or transparent objects. In such cases, single-layer disparity estimation is insufficient, resulting in artifacts and incorrect pixel shifts during warping. Our work bypasses these restrictions by directly synthesizing the new viewpoint, avoiding any intermediate steps. This is achieved by leveraging a pre-trained video model's priors on geometry, object materials, optics, and semantics, without relying on external geometry models or manually disentangling geometry from the synthesis process. We demonstrate the advantages of our approach in complex, real-world scenarios featuring diverse object materials and compositions. See videos on https://video-eye2eye.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。