融合确定性与生成式方法，快速生成高保真新视角图像
UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis
- 用双向Transformer融合多视角图像与射线信息，生成共享表征
- 确定性头处理可见区域，扩散头修复遮挡和未知区域，提升质量
- 端到端训练，渲染速度比纯生成模型快10倍，适合复杂场景
新视角合成（NVS）旨在仅凭少量已知视角的带姿态图像，从任意新相机位姿生成逼真且3D一致的图像。现有确定性网络可快速渲染可见区域，但会模糊未观测区域；而基于扩散的随机方法虽能合理推测缺失内容，但训练和推理成本过高。本文提出一种混合框架，统一两种范式的优点。一个双向Transformer编码多视角图像标记与Plucker-ray嵌入，生成共享隐空间表示。随后两个轻量级头部作用于该表示：(i) 前馈回归头在几何约束充分区域渲染像素；(ii) 遮蔽自回归扩散头完成遮挡或未见区域。整个模型通过联合光度与扩散损失端到端训练，无需人工设计的3D归纳偏置，可在多样场景中良好扩展。实验表明，本方法在图像质量上达到当前最优水平，同时相比完全生成基线将渲染时间降低一个数量级。
原文摘要 · Abstract (English)
Novel view synthesis (NVS) seeks to render photorealistic, 3D-consistent images of a scene from unseen camera poses given only a sparse set of posed views. Existing deterministic networks render observed regions quickly but blur unobserved areas, whereas stochastic diffusion-based methods hallucinate plausible content yet incur heavy training- and inference-time costs. In this paper, we propose a hybrid framework that unifies the strengths of both paradigms. A bidirectional transformer encodes multi-view image tokens and Plucker-ray embeddings, producing a shared latent representation. Two lightweight heads then act on this representation: (i) a feed-forward regression head that renders pixels where geometry is well constrained, and (ii) a masked autoregressive diffusion head that completes occluded or unseen regions. The entire model is trained end-to-end with joint photometric and diffusion losses, without handcrafted 3D inductive biases, enabling scalability across diverse scenes. Experiments demonstrate that our method attains state-of-the-art image quality while reducing rendering time by an order of magnitude compared with fully generative baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。