用视频扩散模型推透明物体深度与法向,零样本超前。
Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation
- 借视频扩散模型内在光学知识,无需标注训练透明物体感知。
- 在真实与合成数据上达零样本最佳性能,深度与法向均更准确稳定。
- 轻量适配器实现高效推理,适合机器人抓取等实时场景应用。
透明物体感知长期困难:折射、反射和透射破坏了立体视觉、飞行时间及纯判别单目深度假设,导致结果有孔洞且时序不稳定。本文观察到现代视频扩散模型已能合成逼真的透明现象,暗示其内化了光学规律。为此构建了名为TransPhy3D的合成视频数据集,包含11,000个序列,使用Blender/Cycles渲染,由类别丰富的静态资产与形状丰富的程序化资产搭配玻璃/塑料/金属材质生成,通过物理光线追踪与OptiX降噪输出RGB、深度与法向图。基于大型视频扩散模型,采用轻量LoRA适配器学习视频到视频的深度(及法向)转换。训练中将RGB与噪声深度潜空间拼接输入DiT主干,联合在TransPhy3D与现有帧级合成数据集上进行共训练,实现任意长度输入视频的时序一致预测。所得模型DKT在包含透明性的实测与合成视频基准(ClearPose、DREDS CatKnown/CatNovel、TransPhy3D-Test)上达到零样本领先表现,优于强基线模型,法向变体在ClearPose上创视频法向估计新纪录。1.3B参数模型每帧仅需约0.17秒。集成至抓取系统后,其深度显著提升对透明、反光与漫反射表面的抓取成功率,超越以往估计器。这些结果共同支持核心主张:‘扩散模型懂透明’。生成式视频先验可被高效、免标注地重用于复杂现实操作任务中的鲁棒、时序一致感知。
原文摘要 · Abstract (English)
Transparent objects remain notoriously hard for perception systems: refraction, reflection and transmission break the assumptions behind stereo, ToF and purely discriminative monocular depth, causing holes and temporally unstable estimates. Our key observation is that modern video diffusion models already synthesize convincing transparent phenomena, suggesting they have internalized the optical rules. We build TransPhy3D, a synthetic video corpus of transparent/reflective scenes: 11k sequences rendered with Blender/Cycles. Scenes are assembled from a curated bank of category-rich static assets and shape-rich procedural assets paired with glass/plastic/metal materials. We render RGB + depth + normals with physically based ray tracing and OptiX denoising. Starting from a large video diffusion model, we learn a video-to-video translator for depth (and normals) via lightweight LoRA adapters. During training we concatenate RGB and (noisy) depth latents in the DiT backbone and co-train on TransPhy3D and existing frame-wise synthetic datasets, yielding temporally consistent predictions for arbitrary-length input videos. The resulting model, DKT, achieves zero-shot SOTA on real and synthetic video benchmarks involving transparency: ClearPose, DREDS (CatKnown/CatNovel), and TransPhy3D-Test. It improves accuracy and temporal consistency over strong image/video baselines, and a normal variant sets the best video normal estimation results on ClearPose. A compact 1.3B version runs at ~0.17 s/frame. Integrated into a grasping stack, DKT's depth boosts success rates across translucent, reflective and diffuse surfaces, outperforming prior estimators. Together, these results support a broader claim: "Diffusion knows transparency." Generative video priors can be repurposed, efficiently and label-free, into robust, temporally coherent perception for challenging real-world manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。