将多光谱图像融入3D高斯点阵,提升重建精度与跨波段信息利用。
Towards Integrating Multi-Spectral Imaging with Gaussian Splatting
- 联合优化多光谱与RGB数据,共享几何结构并建模反射率。
- 新方法使多光谱重建质量显著提升,且改善了原始RGB效果。
- 适用于农业、遥感等需要多光谱3D重建的领域。
本文研究如何将彩色(RGB)和多光谱影像(红、绿、红边、近红外)整合进3D高斯点阵(3DGS)框架,该框架是基于显式辐射场的前沿方法,可从多视角图像快速实现高保真3D重建。尽管3DGS在RGB数据上表现优异,但对额外光谱波段进行独立优化会导致几何不一致,即使真实几何相同。为探究此问题,我们评估三种策略:1)各波段独立重建,无共享结构;2)分步优化,先优化RGB几何,复制后对每个波段同时优化几何与波段表示;3)联合优化,多模态共同优化,可选先进行仅RGB初始化。通过多光谱数据集上的定量指标与新视角渲染结果,验证了联合优化策略的有效性,不仅提升了整体光谱重建质量,还通过光谱交叉作用增强了RGB效果。因此建议将多光谱数据直接融入球谐函数颜色组件,紧凑建模每个高斯的多光谱反射率。此外,分析揭示了引入光谱波段时机与方式的关键权衡,为鲁棒的多模态3DGS重建提供实用指导。
原文摘要 · Abstract (English)
We present a study of how to integrate color (RGB) and multi-spectral imagery (red, green, red-edge, and near-infrared) into the 3D Gaussian Splatting (3DGS) framework, a state-of-the-art explicit radiance-field-based method for fast and high-fidelity 3D reconstruction from multi-view images. While 3DGS excels on RGB data, naive per-band optimization of additional spectra yields poor reconstructions due to inconsistently appearing geometry in the spectral domain. This problem is prominent, even though the actual geometry is the same, regardless of spectral modality. To investigate this, we evaluate three strategies: 1) Separate per-band reconstruction with no shared structure. 2) Splitting optimization, in which we first optimize RGB geometry, copy it, and then fit each new band to the model by optimizing both geometry and band representation. 3) Joint, in which the modalities are jointly optimized, optionally with an initial RGB-only phase. We showcase through quantitative metrics and qualitative novel-view renderings on multi-spectral datasets the effectiveness of our dedicated optimized Joint strategy, increasing overall spectral reconstruction as well as enhancing RGB results through spectral cross-talk. We therefore suggest integrating multi-spectral data directly into the spherical harmonics color components to compactly model each Gaussian's multi-spectral reflectance. Moreover, our analysis reveals several key trade-offs in when and how to introduce spectral bands during optimization, offering practical insights for robust multi-modal 3DGS reconstruction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。