arXiv:2506.19585cs.CV2025-06ICCV被引 17

一个能自适应多种遥感传感器的通用模型,实现跨传感器数据统一处理。

SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images

  • 通过跨传感器令牌混洗训练统一变压器,生成传感器无关特征。
  • 在单模态和多模态任务上均超越依赖特定传感器的模型性能。
  • 支持任意波段组合输入,适合需要灵活处理多源遥感数据的场景。

从光学传感器到微波雷达,利用遥感(RS)传感器的互补优势对于实现地球密集的时空监测至关重要。然而,当前深度学习模型通常针对单一传感器或固定组合设计:适应不同传感器输入需修改架构并重新训练,限制了可扩展性和跨传感器泛化能力。相比之下,能够动态调整特征表示以接受多样传感器输入的单一模型,将推动遥感数据处理的敏捷性与灵活性。为此,我们提出SMARTIES,一种通用且多功能的基础模型,可消除传感器特定/依赖的限制,实现对多种遥感传感器的可扩展与泛化。SMARTIES将异构传感器数据投影至共享的谱感知空间,支持任意波段组合用于训练和推理。为获得传感器无关表示,我们训练单一统一的Transformer模型,通过跨传感器令牌混洗重构掩码多传感器数据。在多种传感器上的单模态与多模态任务中,SMARTIES均优于依赖传感器特定预训练的先前模型。代码与预训练模型已公开于 https://gsumbul.github.io/SMARTIES。

原文摘要 · Abstract (English)

From optical sensors to microwave radars, leveraging the complementary strengths of remote sensing (RS) sensors is crucial for achieving dense spatio-temporal monitoring of our planet. In contrast, recent deep learning models, whether task-specific or foundational, are often specific to single sensors or to fixed combinations: adapting such models to different sensory inputs requires both architectural changes and re-training, limiting scalability and generalization across multiple RS sensors. On the contrary, a single model able to modulate its feature representations to accept diverse sensors as input would pave the way to agile and flexible multi-sensor RS data processing. To address this, we introduce SMARTIES, a generic and versatile foundation model lifting sensor-specific/dependent efforts and enabling scalability and generalization to diverse RS sensors: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space, enabling the use of arbitrary combinations of bands both for training and inference. To obtain sensor-agnostic representations, we train a single, unified transformer model reconstructing masked multi-sensor data with cross-sensor token mixup. On both single- and multi-modal tasks across diverse sensors, SMARTIES outperforms previous models that rely on sensor-specific pretraining. Our code and pretrained models are available at https://gsumbul.github.io/SMARTIES.

遥感多传感器自编码器基础模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。