用惯性视觉联合优化提升头戴设备的逼真场景重建效果
Photoreal Scene Reconstruction from an Egocentric Device
- 通过VIBA校准滚动快门相机的精确时间与运动轨迹
- 结合物理成像模型，使重建在高动态范围下提升1dB PSNR
- 适合关注头戴设备真实感重建的研究者与开发者
本文研究使用头戴设备进行高动态范围下逼真场景重建的挑战。现有方法通常假设基于视觉-惯性里程计系统以帧率估计6自由度位姿，可能忽略像素级重建所需的细节。本研究提出两项关键发现：首先，不同于主流将RGB相机视为全局快门帧率相机的做法，我们强调采用视觉-惯性捆绑调整（VIBA）校准滚动快门RGB传感器在高频轨迹下的精确时间戳与运动，确保滚动快门相机物理特性的准确建模；其次，将基于物理的图像形成模型融入高斯点阵（Gaussian Splatting），有效处理相机的滚动快门效应及传感器动态范围。所提方法适用于广泛使用的高斯点阵变体。我们在开源Project Aria设备及Meta Quest3设备上进行了全面评估，涵盖多种室内外光照条件。所有实验中，引入VIBA使视觉质量稳定提升+1 dB PSNR，进一步结合所提成像模型再提升+1 dB。完整代码、数据集与采集配置已公开于http://www.projectaria.com/photoreal-reconstruction/
原文摘要 · Abstract (English)
In this paper, we investigate the challenges associated with using egocentric devices to photorealistic reconstruct the scene in high dynamic range. Existing methodologies typically assume using frame-rate 6DoF pose estimated from the device's visual-inertial odometry system, which may neglect crucial details necessary for pixel-accurate reconstruction. This study presents two significant findings. Firstly, in contrast to mainstream work treating RGB camera as global shutter frame-rate camera, we emphasize the importance of employing visual-inertial bundle adjustment (VIBA) to calibrate the precise timestamps and movement of the rolling shutter RGB sensing camera in a high frequency trajectory format, which ensures an accurate calibration of the physical properties of the rolling-shutter camera. Secondly, we incorporate a physical image formation model based into Gaussian Splatting, which effectively addresses the sensor characteristics, including the rolling-shutter effect of RGB cameras and the dynamic ranges measured by sensors. Our proposed formulation is applicable to the widely-used variants of Gaussian Splats representation. We conduct a comprehensive evaluation of our pipeline using the open-source Project Aria device under diverse indoor and outdoor lighting conditions, and further validate it on a Meta Quest3 device. Across all experiments, we observe a consistent visual enhancement of +1 dB in PSNR by incorporating VIBA, with an additional +1 dB achieved through our proposed image formation model. Our complete implementation, evaluation datasets, and recording profile are available at http://www.projectaria.com/photoreal-reconstruction/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。