用生成模型将二值单光子图像转为彩色新视角，实现高质量3D重建。
SPC to 3D: Novel View Synthesis from Binary SPC via I2I translation
- 先用Pix2PixHD将二值SPC图像转为彩色图像
- 再用NeRF或3DGS重建3D场景并生成新视角
- 在感知质量和几何一致性上显著优于基线
单光子雪崩二极管（SPAD）是一种前沿成像技术，可实现单光子级检测与高精度时间分辨。基于此，单光子相机（SPC）可在低光和强光条件下实现超高速成像。从这类SPC数据中实现3D重建与辐射场恢复具有重要意义。然而，SPC图像的二值特性导致纹理与色彩信息严重丢失，传统3D合成方法失效。为此，我们提出一种模块化两阶段框架：第一阶段采用Pix2PixHD等生成模型进行图像到图像（I2I）转换，将二值SPC输入映射为合理RGB表示；第二阶段利用神经辐射场（NeRF）或高斯溅射（3DGS）等3D重建技术生成新视角。通过大量定性与定量实验验证，该两阶段流程（Pix2PixHD + NeRF/3DGS）在感知质量与几何一致性上均显著优于基线方案。
原文摘要 · Abstract (English)
Single Photon Avalanche Diodes (SPADs) represent a cutting-edge imaging technology, capable of detecting individual photons with remarkable timing precision. Building on this sensitivity, Single Photon Cameras (SPCs) enable image capture at exceptionally high speeds under both low and high illumination. Enabling 3D reconstruction and radiance field recovery from such SPC data holds significant promise. However, the binary nature of SPC images leads to severe information loss, particularly in texture and color, making traditional 3D synthesis techniques ineffective. To address this challenge, we propose a modular two-stage framework that converts binary SPC images into high-quality colorized novel views. The first stage performs image-to-image (I2I) translation using generative models such as Pix2PixHD, converting binary SPC inputs into plausible RGB representations. The second stage employs 3D scene reconstruction techniques like Neural Radiance Fields (NeRF) or Gaussian Splatting (3DGS) to generate novel views. We validate our two-stage pipeline (Pix2PixHD + Nerf/3DGS) through extensive qualitative and quantitative experiments, demonstrating significant improvements in perceptual quality and geometric consistency over the alternative baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。