用RGB图像生成几何提示,弥补缺失或损坏的深度信息。
GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depth
- 仅用RGB图像生成任务导向的几何提示,替代缺失深度通道。
- 在SUN RGB-D上提升6.1 mIoU,严重损坏时仍增益3.6 mIoU。
- 比单目深度估计快近10倍,适合实时机器人感知应用。
机器人与具身AI的多模态感知系统常依赖可靠的RGB-D数据,但实际中深度信息常缺失、噪声大或损坏。为此,我们提出GeomPrompt,一种轻量级跨模态适配模块,仅通过RGB图像生成面向任务的几何提示,注入冻结的RGB-D语义分割模型第四通道,无需深度监督。进一步提出GeomPrompt-Recovery,通过预测修正项补偿退化深度,增强冻结分割器性能。两者均仅以下游分割监督训练,聚焦恢复对分割有用的几何先验,而非重建深度信号。在SUN RGB-D数据集上,GeomPrompt相较纯RGB推理,使DFormer提升6.1 mIoU,GeminiFusion提升3.0 mIoU,且优于强单目深度估计算法。面对严重深度损坏,GeomPrompt-Recovery持续提升鲁棒性,最高获3.6 mIoU增益。该方法效率显著:延迟仅7.8毫秒,远低于单目深度基线(38.3毫秒和71.9毫秒)。结果表明,任务驱动的几何提示是应对深度缺失与退化的高效机制。
原文摘要 · Abstract (English)
Multimodal perception systems for robotics and embodied AI often assume reliable RGB-D sensing, but in practice, depth is frequently missing, noisy, or corrupted. We thus present GeomPrompt, a lightweight cross-modal adaptation module that synthesizes a task-driven geometric prompt from RGB alone for the fourth channel of a frozen RGB-D semantic segmentation model, without depth supervision. We further introduce GeomPrompt-Recovery, an adaptation module that compensates for degraded depth by predicting the fourth channel correction relevant for the frozen segmenter. Both modules are trained solely with downstream segmentation supervision, enabling recovery of the geometric prior useful for segmentation, rather than estimating depth signals. On SUN RGB-D, GeomPrompt improves over RGB-only inference by +6.1 mIoU on DFormer and +3.0 mIoU on GeminiFusion, while remaining competitive with strong monocular depth estimators. For degraded depth, GeomPrompt-Recovery consistently improves robustness, yielding gains up to +3.6 mIoU under severe depth corruptions. GeomPrompt is also substantially more efficient than monocular depth baselines, reaching 7.8 ms latency versus 38.3 ms and 71.9 ms. These results suggest that task-driven geometric prompting is an efficient mechanism for cross-modal compensation under missing and degraded depth inputs in RGB-D perception.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。