用可微分多平面图像实现快速轻量级新视角合成
Fast and Lightweight Novel View Synthesis with Differentiable Multiplane Image
- 基于可微分优化的多平面图像表示，结合视觉大模型初始化几何
- 在稀疏视图下仍保持高质量渲染，速度比3DGS快30.7%，模型小至14.8%
- 引入一步扩散修复空洞与伪影，适合移动端部署
新视角合成近年取得显著进展，主流方法如神经辐射场（NeRF）和3D高斯溅射（3DGS）虽效果出色，但常难以兼顾渲染速度与模型体积，且优化训练耗时长。这些方法通常依赖密集观测，在稀疏视图下表现不佳。尽管前馈重建大幅降低3DGS的优化时间，其像素对齐结构会从单张图像生成数百万高斯点，严重限制移动端部署。为此，本文重新审视多平面图像（MPI）表示，利用视觉基础模型预测点图实现可靠几何初始化，再进行可微分优化。针对稀疏初始化导致的空洞与伪影问题，提出一步扩散机制，参与MPI的可微分优化与渲染后处理。相比代表性基于GS的方法，本方法速度提升30.7%，模型仅占其14.8%，在正面视图场景下实现媲美的合成质量。
原文摘要 · Abstract (English)
Recently, novel view synthesis has witnessed remarkable progress, with mainstream methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) delivering impressive results. However, these approaches often struggle to balance rendering speed and model size, and their optimization-based training can be highly time-consuming. Furthermore, they typically rely on dense observations, often failing to produce satisfactory results under sparse-view conditions. Although feed-forward reconstruction significantly reduces the optimization time of 3DGS, its pixel-aligned formulation generates millions of Gaussians from a single image, severely limiting its practical deployment on mobile devices. To address these limitations, we revisit the Multiplane Image(MPI) representation, which represents scenes using a compact set of planar layers for efficient novel view synthesis. Leveraging recent advances in visual foundation models, we utilize predicted point maps for reliable geometric initialization, followed by differentiable optimization. To address the issues of holes and artifacts in sparsely initialized MPI, we introduce one-step diffusion, which participates in both the differentiable optimization of MPI and the postprocessing of rendering results. Compared with a representative GS-based method, our approach is 30.7% faster and uses only 14.8% of its model size, while achieving competitive synthesis quality on front-view scenarios
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。