将3D感知注入2D潜在空间，实现逼真3D重建。
Latent Radiance Fields with 3D-aware 2D Representations
- 在2D潜在空间引入3D一致性约束，提升特征表达
- 构建潜伏辐射场，在多场景下实现高保真3D生成
- 首次证明2D潜在表示可生成类辐射场级3D效果，适合3D生成研究者
潜伏3D重建通过将2D特征压缩至3D空间，在3D语义理解与生成中展现出巨大潜力。然而，现有方法在2D特征空间与3D表示之间存在域差异，导致渲染性能下降。为此，我们提出一种新框架，将3D感知融入2D潜在空间。该框架包含三个阶段：(1) 对应关系感知的自编码方法，增强2D潜在表示的3D一致性；(2) 潜伏辐射场（LRF），将这些3D感知的2D表示升维至3D空间；(3) 基于变分自编码器-辐射场（VAE-RF）的对齐策略，提升从渲染2D表示还原图像的性能。大量实验表明，本方法在合成表现与跨数据集泛化能力上优于当前最优潜伏3D重建方法，适用于多样化的室内外场景。据我们所知，这是首个展示由2D潜在表示构建的辐射场可实现照片级3D重建的工作。
原文摘要 · Abstract (English)
Latent 3D reconstruction has shown great promise in empowering 3D semantic understanding and 3D generation by distilling 2D features into the 3D space. However, existing approaches struggle with the domain gap between 2D feature space and 3D representations, resulting in degraded rendering performance. To address this challenge, we propose a novel framework that integrates 3D awareness into the 2D latent space. The framework consists of three stages: (1) a correspondence-aware autoencoding method that enhances the 3D consistency of 2D latent representations, (2) a latent radiance field (LRF) that lifts these 3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field (VAE-RF) alignment strategy that improves image decoding from the rendered 2D representations. Extensive experiments demonstrate that our method outperforms the state-of-the-art latent 3D reconstruction approaches in terms of synthesis performance and cross-dataset generalizability across diverse indoor and outdoor scenes. To our knowledge, this is the first work showing the radiance field representations constructed from 2D latent representations can yield photorealistic 3D reconstruction performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。