研究地理空间大模型在不同数据量下的性能规律,发现大模型+大数据下视觉变压器表现最优。
Scaling Laws for Geospatial Foundation Models: A case study on PhilEO Bench
- 对比三种架构在0.5TB到23TB数据上的表现,系统探索规模效应
- 23TB数据上视觉变压器在语义分割任务领先,200M参数卷积模型在小样本中更优
- 首次评估状态空间模型在遥感领域潜力,代码与数据开源可复现
基础模型通过大规模预训练在多个领域达到顶尖性能。在地球观测(EO)领域,海量卫星数据(拍字节级)推动了地理空间基础模型(GFMs)的发展。然而,数据规模、模型架构和模型大小如何共同影响下游性能仍不明确。本文系统研究这一设计空间,在三个数据规模下进行预训练与微调:PhilEO Globe(0.5TB)、FastTOM(2TB,本文提出)、MajorTOM(23TB)。评估三类架构:基于CNN的Geo-Aware U-Net、基于Transformer的ViT-UPerNet,以及基于状态空间模型的Mamba;模型参数量从44M至300M不等。所有模型均在PhilEO Bench上测试,涵盖道路密度与建筑密度回归、土地覆盖分割任务,并与TerraMind、Prithvi-EO-2.0等现有模型对比。结果表明,小样本场景下卷积模型仍具竞争力,200M参数的Geo-Aware U-Net在回归任务上优于更大架构;而扩展至多拍字节数据时,ViT-UPerNet表现最佳,尤其在MajorTOM(23TB)上的语义分割任务。首次全面评估Mamba在地球观测中的表现,凸显其潜在效率优势,但需更大规模预训练才能超越CNN与ViT。所有代码、预训练模型及FastTOM数据集均已公开,支持可复现性与未来对地理空间模型缩放规律的研究。
原文摘要 · Abstract (English)
Foundation Models (FMs) have achieved state-of-the-art performance across domains by leveraging large-scale pretraining. In Earth Observation (EO), the availability of petabyte-scale satellite archives has recently enabled the development of GeoSpatial Foundation Models (GFMs). Yet, fundamental questions remain regarding how dataset size, model architecture, and size interact to determine downstream performance. In this work, we systematically explore this design space by pretraining and fine-tuning models on three dataset scales: PhilEO Globe (0.5TB), FastTOM (2TB, introduced here), and MajorTOM (23TB). We evaluate three architectural families: Geo-Aware U-Net (CNN), ViT-UPerNet (Transformer), and Mamba (State-Space Model); across model sizes ranging from 44M to 300M parameters. All models are benchmarked on the PhilEO Bench, covering: road density and building density regression, and land cover segmentation, and are compared against existing GFMs such as TerraMind and Prithvi-EO-2.0. Our results show that CNN-based models remain highly competitive in low-shot settings, with a 200M-parameter Geo-Aware U-Net outperforming larger architectures on regression tasks. However, when scaling to multi-terabyte datasets, ViT-UPerNet achieves the best performance, particularly for semantic segmentation on MajorTOM (23TB). Finally, we provide the first extensive evaluation of Mamba models in EO, highlighting their potential efficiency advantages, though further large-scale pretraining is required to fully match CNNs and ViTs. All code, pretrained models, and the FastTOM dataset are released publicly, enabling reproducibility and further exploration of scaling laws for GFMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。