用RGB模型指导热成像深度估计,提升夜间雨天机器人感知能力
Repurposing RGB-based Foundation Model for Depth Estimation on Thermal Images Using Hierarchical Supervision

- 通过双分支结构对齐热图与RGB图像的多层级特征
- 在多个数据集上实现优于基线的深度估计精度
- 适合需要跨模态感知的机器人视觉研究者
从热成像中进行深度估计在夜间、雨天等恶劣环境下对机器人应用极具价值。现有研究尝试将基于RGB的通用模型知识迁移至热成像模态,但其丰富的分层表征仍未被充分挖掘。为此,我们提出一种新框架RGB-HS,利用基于RGB的通用模型提供分层监督来实现热成像深度估计。具体地,将基准热成像编码器替换为通用模型,并引入一个并行的RGB分支,同样使用同构架构的通用模型作为编码器,输入为RGB图像。在两个编码器的多个层级间进行特征对齐,使热成像学生分支能同时获取结构精度和语义抽象能力。此外,我们引入质量验证机制,根据RGB图像质量对对齐过程中的令牌加权。大量实验表明,RGB-HS在主流基准上达到有竞争力的性能,更有效地利用了基于RGB的通用模型在热成像深度估计中的表征潜力。
原文摘要 · Abstract (English)
Depth estimation from thermal images is highly valuable for robotic applications in adverse conditions, such as nighttime and rainy weather. Recent studies have sought to transfer knowledge from RGB-based foundation models to thermal modalities, yet the rich hierarchical representations these models encode remain underutilized. To address this limitation, we propose RGB-HS, a novel framework for thermal-image depth estimation that leverages hierarchical supervision from an RGB-based foundation model. Specifically, we first replace the baseline thermal encoder with a foundational model and introduce a parallel RGB branch that also employs a foundational model as an encoder of the same architecture, taking RGB images as input. The alignment is then performed across multiple levels between the tokens of the two encoders, allowing the thermal student branch to capture both structural precision and semantic abstraction from the RGB teacher branch. Furthermore, we introduce verification to refine the alignment process by weighting tokens from the RGB branch based on RGB image quality. Extensive experiments on the popular benchmark demonstrate that RGB-HS achieves competitive performance and more effectively exploits the representational capacity of RGB-based foundation models for depth estimation on thermal images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。