arXiv:2603.18774cs.CV2026-03被引 1

让视觉几何模型高效融合可见光与热成像,实现高精度3D重建。

SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for Unpaired RGB+Thermal 3D Reconstruction

  • 仅用小规模数据微调,即可让预训练模型适配多模态输入。
  • 在光照不足、浓烟等恶劣条件下仍保持稳定性能,重建细节更丰富。
  • 适合需要跨模态感知的自动驾驶、安防等场景应用。

基于大规模RGB数据预训练的前馈视觉几何模型能有效进行相机位姿估计和场景重建,但其在混合传感模态(如RGB-热成像)上表现下降。我们发现,虽该模型可良好处理纯热成像,但在联合处理时难以对齐可见光与热成像。为此,提出SEAR——一种简单高效的微调策略,使预训练几何变换器适应多模态输入。尽管仅在较小的RGB-T数据集上训练,本方法在3D重建与相机位姿估计上均显著优于现有方法,各项指标均有提升,且推理时间几乎无增加。尤其在低光照、浓烟等挑战性环境下仍能实现可靠重建。通过大量消融实验验证了模型对双模态的有效对齐。此外,我们构建了一个新数据集,包含不同时刻、视角和光照条件下的同步RGB与热成像序列,为未来多模态3D重建研究提供基准。代码与模型已公开于https://doi.org/10.5281/ZENODO.21077295。

原文摘要 · Abstract (English)

Foundational feed-forward visual geometry models enable accurate and efficient camera pose estimation and scene reconstruction by learning strong scene priors from massive RGB datasets. However, their effectiveness drops when applied to mixed sensing modalities, such as RGB-thermal (RGB-T) images. We observe that while a visual geometry grounded transformer pretrained on RGB data generalizes well to thermal-only reconstruction, it struggles to align RGB and thermal modalities when processed jointly. To address this, we propose SEAR, a simple yet efficient fine-tuning strategy that adapts a pretrained geometry transformer to multimodal RGB-T inputs. Despite being trained on a relatively small RGB-T dataset, our approach significantly outperforms state-of-the-art methods for 3D reconstruction and camera pose estimation, achieving significant improvements over all metrics and delivering higher detail and consistency between modalities with negligible overhead in inference time compared to the original pretrained model. Notably, SEAR enables reliable multimodal pose estimation and reconstruction even under challenging conditions, such as low lighting and dense smoke. We validate our architecture through extensive ablation studies and demonstrate how the model aligns both modalities. Additionally, we introduce a new dataset featuring RGB and thermal sequences captured at different times, viewpoints, and illumination conditions, providing a robust benchmark for future work in multimodal 3D scene reconstruction. Code and models are publicly available at https://doi.org/10.5281/ZENODO.21077295.

3D重建多模态热成像视觉几何

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。