用单目图像自监督增强轻量级ToF传感器的深度图
Self-Supervised Enhancement for Depth from a Lightweight ToF Sensor with Monocular Images
- 以自监督方式融合低分辨率深度与高分辨率彩色图
- 在NYU和ScanNet上提升深度图细节与尺度准确性
- 适合资源受限场景下的实时深度补全应用
利用配对的高分辨率彩色图像提升轻量级飞行时间（ToF）传感器输出的低分辨率深度图，是一种经济高效的方案。然而，直接采用传统深度估计流程融合多模态数据需依赖真实深度标签进行监督。为此，我们提出自监督学习框架SelfToF，可生成细节丰富且具备尺度感知能力的深度图。基于图像自监督深度估计流程，引入低分辨率深度作为输入，设计新的深度一致性损失，提出尺度恢复模块，显著提升性能。此外，考虑到真实场景中ToF信号稀疏性差异，进一步升级为SelfToF*，采用子流形卷积与引导特征融合，使模型在不同稀疏程度下均保持稳定表现。大量实验验证了方法的高效性与有效性，测试涵盖NYU和ScanNet数据集。代码已公开于https://github.com/denyingmxd/selftof。
原文摘要 · Abstract (English)
Depth map enhancement using paired high-resolution RGB images offers a cost-effective solution for improving low-resolution depth data from lightweight ToF sensors. Nevertheless, naively adopting a depth estimation pipeline to fuse the two modalities requires groundtruth depth maps for supervision. To address this, we propose a self-supervised learning framework, SelfToF, which generates detailed and scale-aware depth maps. Starting from an image-based self-supervised depth estimation pipeline, we add low-resolution depth as inputs, design a new depth consistency loss, propose a scale-recovery module, and finally obtain a large performance boost. Furthermore, since the ToF signal sparsity varies in real-world applications, we upgrade SelfToF to SelfToF* with submanifold convolution and guided feature fusion. Consequently, SelfToF* maintain robust performance across varying sparsity levels in ToF data. Overall, our proposed method is both efficient and effective, as verified by extensive experiments on the NYU and ScanNet datasets. The code is available at \href{https://github.com/denyingmxd/selftof}{https://github.com/denyingmxd/selftof}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。