无需3D标注，自监督训练实现高质量多视角图像生成
RayZer: A Self-supervised Large View Synthesis Model
- 通过自预测相机位姿实现3D感知的自编码架构
- 在无标注场景下达到甚至超过带真实位姿的方法性能
- 适合对3D重建与图像生成感兴趣的开发者和研究者
我们提出RayZer，一种无需任何3D监督（如相机位姿和场景几何）的自监督多视角3D视觉模型，却展现出显著的3D意识。具体而言，RayZer以未标定、未对齐的图像为输入，自主恢复相机参数，重建场景表示并合成新视图。训练过程中，仅依赖自身预测的相机位姿来渲染目标视图，完全避免使用真实相机标注，仅需2D图像监督即可完成训练。其涌现的3D能力源于两点：一是设计了自监督框架，通过解耦相机与场景表征实现3D感知的自编码；二是采用基于Transformer的模型，唯一引入的3D先验是射线结构，同时连接相机、像素与场景。RayZer在新视图合成任务上表现媲美甚至超越依赖真实位姿的“理想”方法。
原文摘要 · Abstract (English)
We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. Concretely, RayZer takes unposed and uncalibrated images as input, recovers camera parameters, reconstructs a scene representation, and synthesizes novel views. During training, RayZer relies solely on its self-predicted camera poses to render target views, eliminating the need for any ground-truth camera annotations and allowing RayZer to be trained with 2D image supervision. The emerging 3D awareness of RayZer is attributed to two key factors. First, we design a self-supervised framework, which achieves 3D-aware auto-encoding of input images by disentangling camera and scene representations. Second, we design a transformer-based model in which the only 3D prior is the ray structure, connecting camera, pixel, and scene simultaneously. RayZer demonstrates comparable or even superior novel view synthesis performance than ``oracle'' methods that rely on pose annotations in both training and testing. Project: https://hwjiang1510.github.io/RayZer/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。