用稀疏dToF数据生成高精度稠密深度图,适合虚拟现实与机器人应用。
Dense Metric Depth Completion from Sparse Direct Time-of-Flight Sensors

- 双分支Vision Transformer融合图像与稀疏深度,通过掩码注意力精准引导
- 合成数据训练实现零样本跨设备泛化,在6个数据集上超越现有方法
- 无需复杂后处理,轻量解码器高效重建,适合实时系统部署
直接飞行时间(dToF)传感器提供高精度度量深度,在真实场景中比间接ToF更鲁棒。但其制造成本高、光敏阵列小,导致深度图极度稀疏、低分辨率且含噪,难以用于需要稠密度量深度的VR/XR、机器人及三维感知任务。现有单目和深度补全方法难以应对dToF的独特采样模式与硬件伪影,在严重稀疏或噪声条件下性能显著下降。本文提出一个通用的稠密度量深度补全框架,可适配多种传感器类型、稀疏度与噪声条件。模型采用深度引导的双分支Vision Transformer编码器,分别处理RGB图像与稀疏dToF数据;掩码联合注意力模块确保深度令牌可靠引导图像特征而不被覆盖。轻量解码器高效重建稠密度量深度,无需基于扩散或繁重后处理。为缓解配对训练数据稀缺问题,引入全面的dToF模拟管道,重现闪光、亚VGA闪光及旋转传感器的特性,包括硬件退化、不规则稀疏性和真实噪声分布。模型完全在合成数据上训练,实现对6个数据集和3种真实dToF设备的强零样本泛化,精度与计算效率均优于现有先进方法。代码与模型已开源。
原文摘要 · Abstract (English)
Direct Time-of-Flight (dToF) sensors provide highly accurate metric depth and are more robust than indirect ToF systems in challenging real-world conditions. However, their high manufacturing cost and limited photodiode array size produce depth maps that are extremely sparse, low-resolution, and noisy, making them unsuitable for VR/XR, robotics, and 3D perception tasks that require dense metric depth. Existing monocular and depth completion methods struggle to handle the unique sampling patterns and hardware artifacts of dToF devices, and their performance often deteriorates significantly under severe sparsity or noise. We present a generalizable framework for dense metric depth completion from sparse dToF measurements, capable of operating across diverse sensor types, sparsity levels, and noise conditions. Our model employs a depth-guided dual-branch Vision Transformer encoder that processes RGB images and sparse dToF measurements separately, while a masked joint attention module allows depth tokens to reliably guide image features without being overwritten by them. A lightweight decoder reconstructs dense metric depth efficiently, without diffusion-based or refinement-heavy post-processing. To address the scarcity of paired training data, we introduce a comprehensive dToF simulation pipeline that reproduces the characteristics of flash, sub-VGA flash, and rotating sensors, including hardware-induced degradation, irregular sparsity, and realistic noise distributions. Trained entirely on synthetic data, our model achieves strong zero-shot generalization across 6 datasets and 3 real dToF devices, outperforming state-of-the-art approaches in both accuracy and computational efficiency. This establishes a robust and practical solution for dense metric depth completion from sparse direct ToF sensors. Our code and models are open-sourced. See https://vclab.kaist.ac.kr/cvpr2026p3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。