用3D高斯表示融合红外与可见光图像，实现自由视角渲染。
Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation
- 基于多模态2D图像重建3D场景几何，用高斯模型实现跨模态融合。
- 提出交叉调制模块，通过调节高斯透明度解决红外与可见光冲突问题。
- 融合损失函数确保关键特征保留，适合复杂场景的全天候视觉应用。
红外-可见光图像融合旨在将红外与可见光信息整合为单一融合图像。现有二维融合方法仅关注固定视角下的图像融合，难以全面理解复杂场景，导致关键信息丢失。为此，我们提出一种新型红外-可见光高斯融合（IVGF）框架，从多模态二维输入中重建场景几何，并实现融合图像的直接渲染。具体地，我们设计了交叉模态调制（CMA）模块，通过调节高斯的不透明度来解决跨模态冲突问题。此外，为保留双模态的独特特征，引入融合损失函数以指导CMA优化，确保融合图像保留各模态的关键特性。大量定性和定量实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Infrared-visible image fusion aims to integrate infrared and visible information into a single fused image. Existing 2D fusion methods focus on fusing images from fixed camera viewpoints, neglecting a comprehensive understanding of complex scenarios, which results in the loss of critical information about the scene. To address this limitation, we propose a novel Infrared-Visible Gaussian Fusion (IVGF) framework, which reconstructs scene geometry from multimodal 2D inputs and enables direct rendering of fused images. Specifically, we propose a cross-modal adjustment (CMA) module that modulates the opacity of Gaussians to solve the problem of cross-modal conflicts. Moreover, to preserve the distinctive features from both modalities, we introduce a fusion loss that guides the optimization of CMA, thus ensuring that the fused image retains the critical characteristics of each modality. Comprehensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。