arXiv:2411.17984cs.CV2024-11ICCV被引 13

用热传导模拟遥感图像局部关联,提升模型效率与可解释性。

RS-vHeat: Heat Conduction Guided Efficient Remote Sensing Foundation Model

  • 借鉴热传导原理,设计低复杂度算子捕捉遥感图像结构特征。
  • 自监督学习频域分布表示,在10个数据集上显著提升性能。
  • 相比注意力模型,内存减少84%,吞吐量提升2.7倍,适合大规模遥感应用。

遥感基础模型打破了传统任务专用模型的范式,具备更强的多任务扩展能力,但在处理大规模遥感图像时仍面临计算效率低和可解释性差的问题。为此,本文受热传导过程(物理中建模局部热量扩散)启发,首次探索利用热传导的并行计算模型来模拟高分辨率遥感图像中的局部区域相关性,提出高效多模态遥感基础模型RS-vHeat。具体而言,RS-vHeat:1)采用复杂度为$O(N^{1.5})$且具有全局感受野的热传导算子(HCO),在降低计算开销的同时捕捉遥感目标结构信息以指导热扩散;2)通过基于频域分层掩码与多域重建的自监督策略,学习不同场景的频率分布表征;3)在4项任务、10个数据集上显著优于当前最优方法。相较基于注意力的遥感基础模型,内存降低84%,浮点运算量(FLOPs)减少24%,吞吐量提升2.7倍。代码将公开。

原文摘要 · Abstract (English)

Remote sensing foundation models largely break away from the traditional paradigm of designing task-specific models, offering greater scalability across multiple tasks. However, they face challenges such as low computational efficiency and limited interpretability, especially when dealing with large-scale remote sensing images. To overcome these, we draw inspiration from heat conduction, a physical process modeling local heat diffusion. Building on this idea, we are the first to explore the potential of using the parallel computing model of heat conduction to simulate the local region correlations in high-resolution remote sensing images, and introduce RS-vHeat, an efficient multi-modal remote sensing foundation model. Specifically, RS-vHeat 1) applies the Heat Conduction Operator (HCO) with a complexity of $O(N^{1.5})$ and a global receptive field, reducing computational overhead while capturing remote sensing object structure information to guide heat diffusion; 2) learns the frequency distribution representations of various scenes through a self-supervised strategy based on frequency domain hierarchical masking and multi-domain reconstruction; 3) significantly improves efficiency and performance over state-of-the-art techniques across 4 tasks and 10 datasets. Compared to attention-based remote sensing foundation models, we reduce memory usage by 84\%, FLOPs by 24\% and improves throughput by 2.7 times. The code will be made publicly available.

遥感热传导高效模型自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。