arXiv:2412.19483cs.CV2024-12被引 3

用单张压缩图像恢复3D场景,实现快速高质视图合成。

Learning Radiance Fields from a Single Snapshot Compressive Image

  • 将压缩成像物理过程融入NeRF训练,直接从单张图像重建3D结构。
  • 在合成与真实数据上均超越现有方法,支持实时多视角高清渲染。
  • 结合3D高斯点云,提升重建质量与渲染速度,适合动态场景应用。

本文探索了快照压缩成像(SCI)技术从单张时间压缩图像中恢复潜在3D场景结构的潜力。SCI是一种低成本方法,通过使用特殊设计的2D掩码,将高维数据(如光谱或时间信息)编码至单张二维图像中,降低存储与传输需求,并具备潜在隐私保护能力。受此启发,本文提出SCINeRF,将SCI的物理成像过程建模为NeRF训练的一部分,利用其强大的3D场景表示能力恢复编码的3D信息。进一步地,融合流行的3D高斯溅射(3DGS)框架,提出SCISplat,通过显式优化点云为3D高斯表示,提升3D场景重建质量与训练/渲染速度。我们在合成数据和自建SCI系统采集的真实数据上进行了广泛评估。实验结果表明,所提方法在图像重建与新视角合成方面均优于当前最优方法。此外,结合SCI与3DGS的渲染能力,该方法可实现实时高帧率、多视角一致的图像生成。代码将在https://github.com/WU-CVGL/SCISplat发布。

原文摘要 · Abstract (English)

In this paper, we explore the potential of Snapshot Compressive Imaging (SCI) technique for recovering the underlying 3D scene structure from a single temporal compressed image. SCI is a cost-effective method that enables the recording of high-dimensional data, such as hyperspectral or temporal information, into a single image using low-cost 2D imaging sensors. To achieve this, a series of specially designed 2D masks are usually employed, reducing storage and transmission requirements and offering potential privacy protection. Inspired by this, we take one step further to recover the encoded 3D scene information leveraging powerful 3D scene representation capabilities of neural radiance fields (NeRF). Specifically, we propose SCINeRF, in which we formulate the physical imaging process of SCI as part of the training of NeRF, allowing us to exploit its impressive performance in capturing complex scene structures. In addition, we further integrate the popular 3D Gaussian Splatting (3DGS) framework and propose SCISplat to improve 3D scene reconstruction quality and training/rendering speed by explicitly optimizing point clouds into 3D Gaussian representations. To assess the effectiveness of our method, we conduct extensive evaluations using both synthetic data and real data captured by our SCI system. Experimental results demonstrate that our proposed approach surpasses the state-of-the-art methods in terms of image reconstruction and novel view synthesis. Moreover, our method also exhibits the ability to render high frame-rate multi-view consistent images in real time by leveraging SCI and the rendering capabilities of 3DGS. Codes will be available at: https://github.com/WU- CVGL/SCISplat.

3D重建压缩成像神经辐射场高斯溅射

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。