研究自监督遥感模型如何迁移,发现模型表现随任务变化且中间层信息更有效。
How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?

- 分析六种主流遥感自监督模型在不同任务中的迁移能力。
- 中间层特征比最终层更含任务相关信号,且各模型深度特征分布各异。
- 适配策略如解码器设计影响大于模型选择,建议更关注表示特性。
自监督地理空间基础模型(GeoFMs)从遥感数据中学习可迁移表征,但其下游行为难以刻画。本文研究了涵盖联合嵌入、重建和多模态预训练的六种代表性GeoFMs,评估其在分类、回归和分割基准上,不同标签可用性与下游流程下的迁移性能。结果发现模型排名随任务和微调设置而变。层间探测显示,多数情况下任务相关信息在中间Transformer块中更易获取,且各模型呈现显著不同的深度特征分布。在PASTIS与Sen1Floods11的分割案例中,下游适配策略如解码器设计和微调的影响甚至超过模型选择本身,标准密集预测头可能与GeoFMs在深度上的信息组织不匹配。最后,通过CKA分析发现,微调并非均匀地改变模型各层,最强变化集中在ViT块中首个MLP的线性层。这些结果解释了GeoFM排名为何在不同基准上变动,并推动更注重表示特性的评估与适配策略。
原文摘要 · Abstract (English)
Self-supervised geospatial foundation models (GeoFMs) learn transferable representations from remote sensing data, but their downstream behavior is difficult to characterize. We study six representative GeoFMs spanning joint-embedding, reconstruction, and multimodal pretraining families, and evaluate transfer across classification, regression, and segmentation benchmarks under different label availability and downstream pipelines. We find that model rankings change across tasks and adaptation settings. Layerwise probing shows that, in most cases, task-relevant information is more accessible in intermediate transformer blocks compared to final-layer embeddings, and that GeoFMs exhibit distinct depthwise profiles. In segmentation case studies on PASTIS and Sen1Floods11, downstream adaptation settings such as decoder design and fine-tuning can be as impactful as the choice of GeoFM, and standard dense-prediction heads may be poorly aligned with how GeoFMs organize information over depth. Finally, CKA analysis on case studies shows that fine-tuning does not rewrite GeoFMs uniformly across depth, and the strongest changes are localized to the first linear layer of the MLP in ViT blocks. These results help explain why GeoFM rankings shift across benchmarks and motivate more representation-aware evaluation and adaptation strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。