用扩散模型均衡多模态特征,提升遥感分类精度
Balanced Diffusion-Guided Fusion for Multimodal Remote Sensing Classification
- 通过自适应掩码策略平衡多模态数据分布
- 融合CNN/Mamba/Transformer,分层引导特征提取
- 适合遥感图像分类与多模态深度学习研究者
基于深度学习的多模态遥感数据分析方法因能有效整合不同传感器提供的空间、光谱和结构信息而日益流行。近年来,去噪扩散概率模型(DDPM)因其强大的空间-光谱分布建模能力受到关注。然而,预训练多模态DDPM可能导致模态失衡,且如何有效利用扩散特征指导互补特征提取仍是开放问题。为此,本文提出一种平衡扩散引导融合(BDGF)框架,利用多模态扩散特征引导多分支网络进行地物分类。具体地,设计自适应模态掩码策略,促使DDPM获得模态均衡而非以光谱图像主导的数据分布;随后,通过特征融合、组通道注意力与跨注意力机制,使扩散特征分层引导CNN、Mamba和Transformer网络的特征提取;最后,引入互学习策略,通过对齐各子网络的概率熵与特征相似性,增强分支间协作。在四个多模态遥感数据集上的大量实验表明,该方法显著提升分类性能。代码已开源:https://github.com/HaoLiu-XDU/BDGF。
原文摘要 · Abstract (English)
Deep learning-based techniques for the analysis of multimodal remote sensing data have become popular due to their ability to effectively integrate complementary spatial, spectral, and structural information from different sensors. Recently, denoising diffusion probabilistic models (DDPMs) have attracted attention in the remote sensing community due to their powerful ability to capture robust and complex spatial-spectral distributions. However, pre-training multimodal DDPMs may result in modality imbalance, and effectively leveraging diffusion features to guide complementary diversity feature extraction remains an open question. To address these issues, this paper proposes a balanced diffusion-guided fusion (BDGF) framework that leverages multimodal diffusion features to guide a multi-branch network for land-cover classification. Specifically, we propose an adaptive modality masking strategy to encourage the DDPMs to obtain a modality-balanced rather than spectral image-dominated data distribution. Subsequently, these diffusion features hierarchically guide feature extraction among CNN, Mamba, and transformer networks by integrating feature fusion, group channel attention, and cross-attention mechanisms. Finally, a mutual learning strategy is developed to enhance inter-branch collaboration by aligning the probability entropy and feature similarity of individual subnetworks. Extensive experiments on four multimodal remote sensing datasets demonstrate that the proposed method achieves superior classification performance. The code is available at https://github.com/HaoLiu-XDU/BDGF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。