用2D多视角注意力修复VAE隐空间的3D细节,提升稀疏视图重建质量。
Splatent: Splatting Diffusion Latents for Novel View Synthesis
- 在VAE隐空间中通过多视角注意力从2D输入恢复细节
- 在多个基准上达到隐空间辐射场重建新SOTA性能
- 兼容现有前馈框架,适合高保真稀疏视图3D重建
辐射场表示近年来被探索用于扩散模型常用变分自编码器(VAE)的隐空间,该方向具备高效渲染和与扩散流水线无缝集成的优势。然而,这类方法存在根本局限:VAE隐空间缺乏多视角一致性,导致3D重建时纹理模糊、细节丢失。现有方法要么通过微调VAE以缓解问题,但牺牲了重建质量;要么依赖预训练扩散模型恢复细节,又带来幻觉风险。本文提出Splatent,一种基于扩散的增强框架,运行于3D高斯点阵(3DGS)的VAE隐空间之上。核心思路突破传统3D中心范式:不直接在3D空间重建细节,而是通过多视角注意力机制,从输入视图中2D恢复细节。该方法在保持预训练VAE重建质量的同时,实现了真实细节还原。在多个基准测试中,Splatent建立新的隐空间辐射场重建最优性能。进一步实验表明,将其集成至现有前馈框架可一致提升细节保留能力,为高质量稀疏视图3D重建开辟新路径。代码已公开于项目页:https://orhir.github.io/Splatent/
原文摘要 · Abstract (English)
Radiance field representations have recently been explored in the latent space of VAEs that are commonly used by diffusion models. This direction offers efficient rendering and seamless integration with diffusion-based pipelines. However, these methods face a fundamental limitation: The VAE latent space lacks multi-view consistency, leading to blurred textures and missing details during 3D reconstruction. Existing approaches attempt to address this by fine-tuning the VAE, at the cost of reconstruction quality, or by relying on pre-trained diffusion models to recover fine-grained details, at the risk of some hallucinations. We present Splatent, a diffusion-based enhancement framework designed to operate on top of 3D Gaussian Splatting (3DGS) in the latent space of VAEs. Our key insight departs from the conventional 3D-centric view: rather than reconstructing fine-grained details in 3D space, we recover them in 2D from input views through multi-view attention mechanisms. This approach preserves the reconstruction quality of pretrained VAEs while achieving faithful detail recovery. Evaluated across multiple benchmarks, Splatent establishes a new state-of-the-art for VAE latent radiance field reconstruction. We further demonstrate that integrating our method with existing feed-forward frameworks, consistently improves detail preservation, opening new possibilities for high-quality sparse-view 3D reconstruction. Code is available on our project page: https://orhir.github.io/Splatent/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。