用扩散桥模型融合雷达与光学图像,高效去除卫星图云层
Multimodal Diffusion Bridge with Attention-Based SAR Fusion for Satellite Image Cloud Removal
- 直接连接有云与无云图像分布,避免从纯噪声采样
- 在SEN12MS-CR数据集上达到当前最优性能
- 专为遥感设计的双分支架构,融合雷达与光学信息
深度学习在融合合成孔径雷达(SAR)图像以去除光学卫星图像云层方面取得一定进展。近期,扩散模型因其能从无云分布中采样,相比早期方法生成更高质量结果而备受关注。然而,扩散模型通常从纯高斯噪声启动采样,导致采样轨迹复杂,性能受限。此外,现有方法在有效融合SAR与光学数据方面仍显不足。为此,本文提出用于云去除的扩散桥模型(DB-CR),直接连接有云与无云图像分布。同时,设计一种新型多模态扩散桥架构,采用双分支主干网络,包含高效骨干结构和专用跨模态融合模块,可有效提取并融合SAR与光学图像特征。通过将云去除建模为扩散桥问题,并利用该定制化架构,DB-CR在保证高保真度的同时具备计算高效性。在SEN12MS-CR云去除数据集上的实验表明,其性能达到当前最优水平。
原文摘要 · Abstract (English)
Deep learning has achieved some success in addressing the challenge of cloud removal in optical satellite images, by fusing with synthetic aperture radar (SAR) images. Recently, diffusion models have emerged as powerful tools for cloud removal, delivering higher-quality estimation by sampling from cloud-free distributions, compared to earlier methods. However, diffusion models initiate sampling from pure Gaussian noise, which complicates the sampling trajectory and results in suboptimal performance. Also, current methods fall short in effectively fusing SAR and optical data. To address these limitations, we propose Diffusion Bridges for Cloud Removal, DB-CR, which directly bridges between the cloudy and cloud-free image distributions. In addition, we propose a novel multimodal diffusion bridge architecture with a two-branch backbone for multimodal image restoration, incorporating an efficient backbone and dedicated cross-modality fusion blocks to effectively extract and fuse features from synthetic aperture radar (SAR) and optical images. By formulating cloud removal as a diffusion-bridge problem and leveraging this tailored architecture, DB-CR achieves high-fidelity results while being computationally efficient. We evaluated DB-CR on the SEN12MS-CR cloud-removal dataset, demonstrating that it achieves state-of-the-art results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。