arXiv:2409.15132cs.CVeess.IV2024-09被引 6

无需预处理,融合多光谱与全色图像重建高保真卫星地表。

FusionRF: High-Fidelity Satellite Neural Radiance Fields from Multispectral and Panchromatic Acquisitions

  • 在优化过程中联合融合多光谱与全色图像,避免传统上采样引入的偏差。
  • 在WorldView-3数据上实现深度重建误差平均降低17%,视图更清晰。
  • 适合遥感、地理信息与高精度三维建模领域的研究人员使用。

我们提出FusionRF,一种从卫星多光谱与全色图像中进行数字表面重建的新框架。现有方法显示神经摄影测量法相比传统算法在光学卫星图像重建上更具精度。常见卫星同时获取多光谱(高光谱分辨率)与全色(高空间分辨率)图像。当前神经重建方法需用全色图像对多光谱图像进行上采样(即去模糊化),但该过程可能因域差距引入偏差与幻觉。FusionRF通过新颖的跨分辨率核,在优化阶段联合融合两模态图像,学习补偿多光谱图像的空间分辨率损失。输入为原始多光谱与全色数据,无需任何预处理。FusionRF还利用多模态外观嵌入,将各模态与视角特征统一编码。通过联合优化双模态,实现图像融合与重建同步完成,省去去模糊化步骤。我们在WorldView-3卫星在多个地点采集的数据上评估该方法,结果表明,相比基线模型,深度重建误差平均降低17%,并能渲染出清晰的训练视图与新视角。

原文摘要 · Abstract (English)

We introduce FusionRF, a novel framework for digital surface reconstruction from satellite multispectral and panchromatic images. Current work has demonstrated the increased accuracy of neural photogrammetry for surface reconstruction from optical satellite images compared to algorithmic methods. Common satellites produce both a panchromatic and multispectral image, which contain high spatial and spectral information respectively. Current neural reconstruction methods require multispectral images to be upsampled with a pansharpening method using the spatial data in the panchromatic image. However, these methods may introduce biases and hallucinations due to domain gaps. FusionRF introduces joint image fusion during optimization through a novel cross-resolution kernel that learns to resolve spatial resolution loss present in multispectral images. As input, FusionRF accepts the original multispectral and panchromatic data, eliminating the need for image preprocessing. FusionRF also leverages multimodal appearance embeddings that encode the image characteristics of each modality and view within a uniform representation. By optimizing on both modalities, FusionRF learns to fuse image modalities while performing reconstruction tasks and eliminates the need for a pansharpening preprocessing step. We evaluate our method on multispectral and panchromatic satellite images from the WorldView-3 satellite in various locations, and show that FusionRF provides an average of 17% reduction in depth reconstruction error, and renders sharp training and novel views.

卫星重建多模态融合神经辐射场

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。