用纯数据驱动方法实现高精度多视角合成,无需3D先验知识。
LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
- 基于Transformer设计双架构,避免传统3D表示与网络结构的先验假设。
- 解码器仅模型在多个数据集上达到最优,比之前最佳方法提升1.5-3.5dB PSNR。
- 仅需1-2张GPU即可超越以往方法,适合资源受限场景下的快速部署。
我们提出大型视图合成模型(LVSM),一种基于Transformer的新型可扩展、泛化性强的稀疏视角输入下新视角合成方法。引入两种架构:(1) 编码器-解码器型LVSM,将输入图像标记编码为固定数量的一维潜在标记,作为全学习的场景表示,并从中解码新视角图像;(2) 仅解码器型LVSM,直接将输入图像映射到新视角输出,彻底消除中间场景表示。两种模型均规避了以往方法中的3D归纳偏置——包括3D表示(如NeRF、3DGS)和网络设计(如对极投影、平面扫描)——以完全数据驱动的方式解决新视角合成问题。尽管编码器-解码器模型因独立潜变量表示而推理更快,但仅解码器型LVSM在质量、可扩展性和零样本泛化能力上表现更优,相较先前最先进方法提升1.5至3.5 dB PSNR。在多个数据集上的全面评估表明,两种LVSM变体均达到新视角合成的顶尖水平。值得注意的是,我们的模型即使在计算资源减少(1-2张GPU)的情况下也超越所有先前方法。
原文摘要 · Abstract (English)
We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, functioning as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods -- from 3D representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps) -- addressing novel view synthesis with a fully data-driven approach. While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalability, and zero-shot generalization, outperforming previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve state-of-the-art novel view synthesis quality. Notably, our models surpass all previous methods even with reduced computational resources (1-2 GPUs). Please see our website for more details: https://haian-jin.github.io/projects/LVSM/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。