统一处理多分辨率多模态遥感数据的视觉变换器模型
UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation
- 用通用图像块编码器统一处理不同分辨率、波段和传感器的数据
- 在多个遥感基准测试上实现强分类与分割性能
- 适合需要跨传感器、跨尺度分析的遥感研究者
视觉变压器（ViT）主导计算机视觉领域，但其依赖固定图像块投影器，难以适配遥感（EO）中多变的空间、光谱和时间分辨率及多种传感器输入。我们提出UniverSat，一种基于通用图像块编码器的ViT类骨干网络，可将来自任意空间、光谱和时间分辨率，以及光学与非光学传感器的图像块，映射到共享嵌入空间，并使用统一权重。该设计支持通过自监督学习在一个异构多模态数据集上训练单一模型，生成鲁棒且传感器无关的空间特征。我们在GeoBench、PANGEABench和SpectralEarth等标准遥感基准上验证了该方法的有效性，表现优异。代码与模型已开源：https://github.com/gastruc/UniverSat。
原文摘要 · Abstract (English)
Vision Transformers (ViT) dominate computer vision. However, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely. We introduce UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions, and from both optical and non-optical sensors, into a shared embedding space with a shared set of weights. This enables training a single model on heterogeneous multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features. We validate this approach with strong results across classification and segmentation on standard EO benchmarks from GeoBench, PANGEABench, and SpectralEarth. Our code and models are available at https://github.com/gastruc/UniverSat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。