融合可见光与热成像,实现更鲁棒的3D场景重建。
ThermoSplat: Cross-Modal 3D Gaussian Splatting with Feature Modulation and Geometry Decoupling
- 通过动态特征调制,让热成像指导可见光纹理生成。
- 独立处理双模态几何差异,热成像单独渲染提升精度。
- 适合需要全天候感知的自动驾驶、安防等场景。
多模态场景重建融合可见光与热红外数据,对复杂光照和天气下的环境感知至关重要。然而,将3D高斯溅射(3DGS)扩展到多光谱场景仍具挑战。现有方法常忽视跨模态相关性,或依赖固定共享表示,难以适应不同波段间的结构关联与物理差异。为此,我们提出ThermoSplat,通过主动特征调制与自适应几何解耦,实现深度光谱感知重建。首先,引入光谱感知自适应调制,动态利用热成像结构先验调节共享隐空间特征,有效引导可见光纹理合成。其次,为应对模态特异性几何不一致,提出模态自适应几何解耦机制,为热分支学习独立透明度偏移并执行独立光栅化。此外,采用混合渲染管线,结合显式球谐函数与隐式神经解码,兼顾语义一致性与高频细节保留。在RGBT-Scenes数据集上的大量实验表明,ThermoSplat在可见与热波段均达到当前最优渲染质量。
原文摘要 · Abstract (English)
Multi-modal scene reconstruction integrating RGB and thermal infrared data is essential for robust environmental perception across diverse lighting and weather conditions. However, extending 3D Gaussian Splatting (3DGS) to multi-spectral scenarios remains challenging. Current approaches often struggle to fully leverage the complementary information of multi-modal data, typically relying on mechanisms that either tend to neglect cross-modal correlations or leverage shared representations that fail to adaptively handle the complex structural correlations and physical discrepancies between spectrums. To address these limitations, we propose ThermoSplat, a novel framework that enables deep spectral-aware reconstruction through active feature modulation and adaptive geometry decoupling. First, we introduce a Spectrum-Aware Adaptive Modulation that dynamically conditions shared latent features on thermal structural priors, effectively guiding visible texture synthesis with reliable cross-modal geometric cues. Second, to accommodate modality-specific geometric inconsistencies, we propose a Modality-Adaptive Geometric Decoupling scheme that learns independent opacity offsets and executes an independent rasterization pass for the thermal branch. Additionally, a hybrid rendering pipeline is employed to integrate explicit Spherical Harmonics with implicit neural decoding, ensuring both semantic consistency and high-frequency detail preservation. Extensive experiments on the RGBT-Scenes dataset demonstrate that ThermoSplat achieves state-of-the-art rendering quality across both visible and thermal spectrums.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。