将NeRF引入潜在空间，实现更快更高质量的3D场景重建。
Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder
- 用3D几何信息约束图像自编码器，使潜在空间对齐3D场景。
- 在潜在空间训练的NeRF质量更高，训练和渲染速度提升约2.5倍。
- 适合需要高效3D生成与跨模型协同的研究者使用。
尽管预训练图像自编码器在计算机视觉中应用日益广泛，但将逆图形学应用于二维潜在空间仍研究不足。然而，将逆图形学引入潜在空间不仅能降低训练与渲染复杂度，还能实现与其他潜在空间方法的互操作性。主要挑战在于，潜在空间缺乏底层3D几何结构，无法直接应用逆图形学。本文提出逆图形自编码器（IG-AE），通过将图像自编码器的潜在空间与联合训练的3D潜在场景对齐，解决此问题。我们基于IG-AE构建了潜在空间的NeRF训练流程，并在Nerfstudio开源框架扩展中实现，从而为支持的方法解锁潜在场景学习能力。实验表明，使用IG-AE训练的潜在NeRF在质量上优于标准自编码器，且相较图像空间中的NeRF，训练与渲染速度提升约2.5倍。项目页面见 https://ig-ae.github.io。
原文摘要 · Abstract (English)
While pre-trained image autoencoders are increasingly utilized in computer vision, the application of inverse graphics in 2D latent spaces has been under-explored. Yet, besides reducing the training and rendering complexity, applying inverse graphics in the latent space enables a valuable interoperability with other latent-based 2D methods. The major challenge is that inverse graphics cannot be directly applied to such image latent spaces because they lack an underlying 3D geometry. In this paper, we propose an Inverse Graphics Autoencoder (IG-AE) that specifically addresses this issue. To this end, we regularize an image autoencoder with 3D-geometry by aligning its latent space with jointly trained latent 3D scenes. We utilize the trained IG-AE to bring NeRFs to the latent space with a latent NeRF training pipeline, which we implement in an open-source extension of the Nerfstudio framework, thereby unlocking latent scene learning for its supported methods. We experimentally confirm that Latent NeRFs trained with IG-AE present an improved quality compared to a standard autoencoder, all while exhibiting training and rendering accelerations with respect to NeRFs trained in the image space. Our project page can be found at https://ig-ae.github.io .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。