用单张RGB-D图生成透明物体精确深度图,实现无需材料先验的抓取。
TransDiff: Diffusion-Based Method for Manipulating Transparent Objects Using a Single RGB-D Image
- 基于扩散模型,融合语义、边缘与法向特征引导深度生成。
- 在合成与真实场景中均显著优于基线,深度误差降低27%以上。
- 适合桌面环境下透明物体操作,尤其适用于无材料先验场景。
由于反射与折射特性,透明物体的操纵面临巨大挑战,严重干扰其三维形状的准确估计。为此,我们提出一种基于单视角RGB-D的深度补全框架TransDiff,利用去噪扩散概率模型(DDPM)实现桌面环境中无需材料先验的物体抓取。具体地,通过提取RGB图像中的语义分割、边缘图和法向图等特征,作为条件来引导深度图生成。方法学习一种迭代去噪过程,将随机深度分布逐步优化为精确深度图,初始深度信息辅助提升透明物体区域的估计精度。此外,提出一种新颖的训练策略,使噪声深度与RGB特征更好地对齐,从而分步优化深度估计。最后,采用改进的推理流程加速去噪过程。大量实验验证表明,该方法在合成与真实世界基准上均显著优于基线,在可接受推理时间内实现更优性能。演示视频见https://wang-haoxiao.github.io/TransDiff/
原文摘要 · Abstract (English)
Manipulating transparent objects presents significant challenges due to the complexities introduced by their reflection and refraction properties, which considerably hinder the accurate estimation of their 3D shapes. To address these challenges, we propose a single-view RGB-D-based depth completion framework, TransDiff, that leverages the Denoising Diffusion Probabilistic Models(DDPM) to achieve material-agnostic object grasping in desktop. Specifically, we leverage features extracted from RGB images, including semantic segmentation, edge maps, and normal maps, to condition the depth map generation process. Our method learns an iterative denoising process that transforms a random depth distribution into a depth map, guided by initially refined depth information, ensuring more accurate depth estimation in scenarios involving transparent objects. Additionally, we propose a novel training method to better align the noisy depth and RGB image features, which are used as conditions to refine depth estimation step by step. Finally, we utilized an improved inference process to accelerate the denoising procedure. Through comprehensive experimental validation, we demonstrate that our method significantly outperforms the baselines in both synthetic and real-world benchmarks with acceptable inference time. The demo of our method can be found on https://wang-haoxiao.github.io/TransDiff/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。