用3D高斯点云实现红外与可见光协同建模,提升多模态场景重建质量。
MrGS: Multi-modal Radiance Fields with 3D Gaussian Splatting for RGB-Thermal Novel View Synthesis
- 通过正交特征提取分离可见光与红外信息,结合视角依赖/独立嵌入策略。
- 融合热传导与辐射定律,使红外渲染更符合物理规律,减少高斯点数量。
- 适合需精确红外建模的自动驾驶、安防监控等多模态视觉应用。
神经辐射场(NeRF)与3D高斯点云(3DGS)在可见光场景重建中表现优异,但多模态渲染尤其是红外热成像仍研究不足。现有方法常忽略热传导与朗伯反射特性。本文提出MrGS,基于3DGS的多模态辐射场,可同时重建RGB与热红外3D场景。具体地,从单一外观特征中通过正交提取获取双模态信息,并根据朗伯反射程度采用视图相关或无关嵌入。进一步,引入两个物理先验:首先在α混合前融入傅里叶热传导定律,模拟邻近高斯点间的热扩散;其次结合斯特藩-玻尔兹曼定律与反平方定律,构建深度感知的热辐射图,为热渲染施加几何约束。实验表明,MrGS在保持高保真度的同时显著减少所需高斯点数。
原文摘要 · Abstract (English)
Recent advances in Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) have achieved considerable performance in RGB scene reconstruction. However, multi-modal rendering that incorporates thermal infrared imagery remains largely underexplored. Existing approaches tend to neglect distinctive thermal characteristics, such as heat conduction and the Lambertian property. In this study, we introduce MrGS, a multi-modal radiance field based on 3DGS that simultaneously reconstructs both RGB and thermal 3D scenes. Specifically, MrGS derives RGB- and thermal-related information from a single appearance feature through orthogonal feature extraction and employs view-dependent or view-independent embedding strategies depending on the degree of Lambertian reflectance exhibited by each modality. Furthermore, we leverage two physics-based principles to effectively model thermal-domain phenomena. First, we integrate Fourier's law of heat conduction prior to alpha blending to model intensity interpolation caused by thermal conduction between neighboring Gaussians. Second, we apply the Stefan-Boltzmann law and the inverse-square law to formulate a depth-aware thermal radiation map that imposes additional geometric constraints on thermal rendering. Experimental results demonstrate that the proposed MrGS achieves high-fidelity RGB-T scene reconstruction while reducing the number of Gaussians.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。