欧洲数据让地理大模型表现更好，关键在光谱多样性。
Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance
- 用全球与洲级数据集训练，对比下游表现差异。
- 欧洲数据在多场景下优于全局和单一大陆数据。
- 光谱多样性是决定性能的关键因素，适合遥感研究者。
新地理空间基础模型引入了新型架构和预训练数据集，常基于不同数据多样性理念采样。性能差异多归因于模型架构或输入模态，而预训练数据的作用很少被研究。为填补此空白，我们系统研究了预训练数据地理构成对下游性能的影响。构建了全球及按洲划分的预训练数据集，并在全局与按洲划分的下游数据集上评估。发现欧洲预训练数据在全局与本地任务中均优于全球及单洲数据。进一步分析10个预训练数据集在大陆、生物群落、地表覆盖和光谱值上的多样性，发现仅光谱多样性与性能强相关，其余弱相关。这一发现确立了构建高性能预训练数据集的新维度。代码、7个新数据集、预训练模型及实验框架已开源至https://github.com/kerner-lab/pretrain-where。
原文摘要 · Abstract (English)
New geospatial foundation models introduce a new model architecture and pretraining dataset, often sampled using different notions of data diversity. Performance differences are largely attributed to the model architecture or input modalities, while the role of the pretraining dataset is rarely studied. To address this research gap, we conducted a systematic study on how the geographic composition of pretraining data affects a model's downstream performance. We created global and per-continent pretraining datasets and evaluated them on global and per-continent downstream datasets. We found that the pretraining dataset from Europe outperformed global and continent-specific pretraining datasets on both global and local downstream evaluations. To investigate the factors influencing a pretraining dataset's downstream performance, we analysed 10 pretraining datasets using diversity across continents, biomes, landcover and spectral values. We found that only spectral diversity was strongly correlated with performance, while others were weakly correlated. This finding establishes a new dimension of diversity to be accounted for when creating a high-performing pretraining dataset. We open-sourced 7 new pretraining datasets, pretrained models, and our experimental framework at https://github.com/kerner-lab/pretrain-where.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。