用单光子传感器重建模糊场景,实现高精度3D建模与着色。
PhotonSplat: 3D Scene Reconstruction and Colorization from SPAD Sensors
- 直接处理SPAD生成的二值图像,融合3D空间滤波降噪
- 支持无参考和有参考两种着色方式,可从单张模糊图恢复色彩
- 适用于动态场景,适合做物体检测与外观编辑的下游任务
基于神经渲染的3D重建技术虽已取得高质量成果,但在相机或物体快速运动导致的运动模糊情况下表现不佳。本文提出PhotonSplat框架,利用单光子雪崩二极管(SPAD)阵列这一高速成像技术,直接从SPAD生成的二值图像中重建3D场景,有效应对噪声与模糊之间的权衡问题。该方法引入一种新型3D空间滤波技术以降低渲染噪声,并支持基于生成先验的无参考着色以及基于参考图像的有参考着色,从而实现分割、目标检测与外观编辑等下游应用。此外,模型扩展至动态场景表示,可处理运动物体。本文还构建了真实世界多视角的PhotonScenes数据集,使用SPAD传感器采集。
原文摘要 · Abstract (English)
Advances in 3D reconstruction using neural rendering have enabled high-quality 3D capture. However, they often fail when the input imagery is corrupted by motion blur, due to fast motion of the camera or the objects in the scene. This work advances neural rendering techniques in such scenarios by using single-photon avalanche diode (SPAD) arrays, an emerging sensing technology capable of sensing images at extremely high speeds. However, the use of SPADs presents its own set of unique challenges in the form of binary images, that are driven by stochastic photon arrivals. To address this, we introduce PhotonSplat, a framework designed to reconstruct 3D scenes directly from SPAD binary images, effectively navigating the noise vs. blur trade-off. Our approach incorporates a novel 3D spatial filtering technique to reduce noise in the renderings. The framework also supports both no-reference using generative priors and reference-based colorization from a single blurry image, enabling downstream applications such as segmentation, object detection and appearance editing tasks. Additionally, we extend our method to incorporate dynamic scene representations, making it suitable for scenes with moving objects. We further contribute PhotonScenes, a real-world multi-view dataset captured with the SPAD sensors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。